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ata  drives  decisions.  This  statement  may  sound  too  simplistic,  but  the  truth 
remains  that  humans  need,  crave,  and  desire  data.  Data  is  used  in  nearly  every 
choice  we  make,  from  the  most  mundane  decisions  such  as  where  to  dine  (location, 
taste,  caloric  content,  type  of  food,  price)  to  the  selection  of  a  fabric  softener  (aroma, 
effectiveness,  volume,  eco-friendliness,  price). 

Data  inundation  and  “analysis  paralysis”  are  real  dangers  due  to  the  ease  of  access 
and  abundance  of  information.  Additionally,  the  mobility  of  personal  computing 
devices  creates  a  data  wave,  cresting  at  our  fingertips.  It  can  overwhelm  any  person,  anywhere 
on  the  planet,  at  any  time. 

Data  enters  our  lives  at  breakneck  speeds  and  frequencies.  Your  commute  to  work  likely 
involves  various  data  deliverables:  AM/FM  radio  broadcasts,  satellite  transmissions,  billboard 
advertisements,  traffic  signs  and  signals,  marquee  commercials,  and  the  odd  gesture  from  a  fel¬ 
low  commuter. 

The  real  data  feast  begins  when  we  arrive  at  work  and  begin  interfacing  with  faxes,  e-mails, 
text  messages,  telephone  calls,  status  reports,  market  forecasts,  meeting  charts,  figures,  facts,  the 
rumor  mill,  and  even  simple  co-worker  chitchat.  What  is  a  person  to  do  with  it  all?  Many  of  us 
will  begin  to  filter  based  upon  priority,  but  this  presents  a  danger  of  filtering  too  much  or 
becoming  overwhelmed  due  to  insufficient  filtering.  We  must  learn  to  manage  and  leverage  data 
effectively  to  be  successful  in  today’s  business  environment  as  well  as  in  our  daily  lives.  The 
December  issue  of  CROSSTALK  is  here  to  help  software  professionals  make  sense  of  it  all. 

In  Dr.  Joseph  P.  Avery’s  article,  A  Different  Kind  of  Web-Based  Knowledge  Management:  The 
DTRA  Acquisition  ToolBook ,  he  shares  the  Defense  Threat  Reduction  Agency’s  method  of  intel¬ 
ligent  storage  and  timely  dissemination  of  data — to  the  right  people  at  the  right  time  via  the 
Web. 

Sandy  Schwalb  offers  assistance  if  you  suffer  from  a  lack  of  qualified  data.  Her  article,  The 
Defense  Technical  Information  Center:  Information  for  the  Defense  Community ,  provides  compelling  rea¬ 
sons  for  utilizing  this  virtual  treasure  trove  of  structured,  vetted,  and  certified  information. 

Equally  important  to  data  is  time  and  money,  and  two  articles  explore  the  use  of  Earned 
Value  Management  (EVM)  data  to  ensure  both  are  optimized  for  project  success.  Through  real 
project  data,  Walt  Lipke  assists  software  professionals  by  analyzing  the  predictive  capabilities  of 
EVM  techniques  in  Project  Duration  Forecasting:  Comparing  Earned  I  blue  Management  Methods  to 
Earned  Schedule.  In  The  Two  Most  Useful  Earned  l  blue  Metrics:  The  CPI  and  the  TCPI,  Quentin  W 
Fleming  and  Joel  M.  Koppelman  describe  die  use  of  EVM  data  as  a  tool  to  predict  needed  per¬ 
formance  levels  for  achieving  financial  success. 

This  issue  also  addresses  ways  to  gain  customers  and  then  keep  them  happy  through  pro¬ 
ducing  quality  products.  In  Certifications  Help  Organisations  and  Clients ,  longtime  CROSSTALK 
contributor  George  Jackelen  offers  insights,  for  requestors  and  prospective  bidders  alike,  as  to 
the  value  of  certifications  in  government  Requests  for  Proposals.  In  Using  Software  Quality 
Methods  to  Keduce  Cost  and  Prevent  Defects,  Rick  Spiewak  and  Karen  McRitchie  offer  a  proactive  and 
practical  best  practices  framework  for  software  construction,  leading  to  better  results,  lower 
costs,  and  less  developmental  mishaps. 

In  die  spirit  of  the  holidays,  please  accept  the  December  issue  as  our  gift  to  you  from  the 
CROSSTALK  family.  We  hope  it  assists  in  making  2009  die  best  year  ever,  and  we  wish  you  and 
your  family  peace  and  prosperity  for  years  to  come. 
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A  Different  Kind  ofWeb-Based  Knowledge  Management: 
The  DTRA  Acquisition  ToolBook 


Dr.  Joseph  P.  Avery 
Defense  Threat  Reduction  Agency 

Knott' ledge  management  (KM)  is  composed  of  practices  deployed  by  organisations  to  identify,  create,  represent,  classify,  and 
disseminate  knowledge  for  reuse,  awareness,  and  learning  to  the  ben  fit  of  information  users.  This  article  demonstrates  the 
practical  integration  of  the  principles  of  KM  and  business  processes  by  the  Defense  Threat  Reduction  Agency  (DTRA).  In 
this  example,  a  technology-centric  approach  to  knowledge  sharing  and  utilisation  was  adopted  to  design  a  sinrple  Web-based 
system  that  provided  highly  needed  “how-to”  and  reference  information  to  DTFL4  acquisition  professionals.  The  Acquisition 
ToolBook  s  successful  development  and  deployment  effectively  integrated  KM  with  internal  process  management,  and  it  per¬ 
mitted  a  systemic  review  of  the  acquisition  process  for  ineffective  procedures  and  policies. 


he  idea  for  the  Web-based  DTRA 
Acquisition  ToolBook1  originated  as  a 
result  of  an  information  environment 
characterized  by  acquisition  task  and 
process  information  that  was  scattered 
throughout  a  myriad  of  DTRA  Web  sites 
as  well  as  shared  and  private  drives — or 
the  information  was  simply  not  available 
in  any  capacity.  This  unfavorable  environ¬ 
ment  was  exacerbated  by  the  DTRA  being 
the  merged  product  of  five  different 
defense  agencies  and  programs.  Like  simi¬ 
lar  government  offices,  the  DTRA  was  a 
hotbed  of  hide-and-seek  information 
hoarding  that  was  not  conducive  to  effi¬ 
cient  acquisition  operations.  Searching  for 
acquisition  data  was  becoming  so  difficult 
and  time-consuming  that  it  periodically 
exceeded  the  anticipated  time  for  actual 
task  completion.  The  DTRA  had  to  devel¬ 
op  a  single  and  easily  accessible,  central¬ 
ized,  and  functionally  based  repository  of 
approved  information,  documentation, 
procedures,  references,  and  processes.  As 
well,  it  had  to  be  available  to  all  acquisition 
professionals,  on  a  single  page,  located  on 
the  agency’s  main  portal. 

The  ToolBook  is  not  a  large,  DoD- 
wide  acquisition  system  such  as  the  previ¬ 
ous  Acquisition  Deskbook,  the  current 
Acquisition  Knowledge  Sharing  System, 
or  the  future  Big  A2  DoD  Acquisition 
Portal.  Those  Big  A  portals  serve  a  broad¬ 
er  purpose  of  acting  as  comprehensive 
repositories  of  acquisition  information 
and  collaboration.  In  the  trenches,  though, 
project  managers  are  looking  for  smaller, 
simpler,  and  faster  portals  of  information 
that  quickly  offer  a  how-to  and  the  refer¬ 
ence  information  needed  to  perform  the 
various  complex  acquisition  tasks  without 
extensive  data  mining  and  infinite  search 
activities.  The  ToolBook  information 
environment  was  designed  to  make  infor¬ 
mation  easily  found  and  accessed  through 
a  single  location  on  the  agency-level  enter¬ 


prise  information  system.  More  impor¬ 
tantly,  this  micro-level  site  provides 
important  agency-specific  acquisition 
information  and  processes.  The  ToolBook 
serves  as  the  agency’s  graphic  interface, 
portraying  the  entire  agency  acquisition 
process  represented  through  24  activity 
boxes  of  related  acquisition  information 
and  tasks.  Every  government  agency 
could  easily  have  a  similar  system. 

The  Blueprint  to  an  Effective 
Information  Environment 

The  ToolBook  not  only  provides  needed 
acquisition  information,  but  it  also 
includes  a  detailed  process  map  of 
required  acquisition  tasks  to  guide  the 
agency’s  acquisition  professionals.  Smaller, 
simpler,  and  faster  were  the  hallmarks  of 
the  successful  acquisition  portal.  In  the 
ToolBook’s  development,  users  new  to 
defense  acquisition  tested  the  system.  Their 
positive  feedback  proved  what  the  DTRA 
was  aiming  for:  The  ToolBook  was  suc¬ 
cessful  in  walking  them  through  the  tasks 
and  activities  required  of  the  acquisition 
effort,  and  then  gave  them  the  tools, 
examples,  forms,  and  references  to  do  the 
job.  In  elevating  the  utility  of  KM,  the 
ToolBook  integrates  both  an  extensive 
library  of  easily  accessible  acquisition 
how-to  and  reference  information  with 
acquisition  processes  and  procedures.  It 
merged  the  what  you  need  to  do  with  the  how 
and  when  to  do  it  information. 

The  ToolBook  is  based  upon  an  inte¬ 
gration  of  Microsoft  SharePoint,  Adobe 
Flash,  and  Microsoft  .NET  application 
software  tied  to  a  Structured  Query 
Language  data  server.  This  combination 
facilitates  simplicity,  speed  of  access,  and 
use,  and  provides  system  flexibility  with  a 
broad  array  of  technical  features  beneficial 
to  system  users.  By  preventing  infinite 
search  activities,  the  ToolBook  improves 
the  speed  and  effectiveness  of  the  user’s 


acquisition  task  completion.  The  critical 
acquisition  information  provided  by  the 
ToolBook  was  tailored  to  meet  the  infor¬ 
mation  needs  of  program  and  project  man¬ 
agers.  However,  it  also  benefits  contracting 
officer  representatives,  contract  specialists, 
and  program  analysts  by  assisting  them  in 
the  performance  of  their  specific  acquisi¬ 
tion  and  procurement  functions. 

The  ToolBook  represents  a  merger  of 
KM,  process  management,  and  opera¬ 
tional  simplicity.  Personnel  cannot  access 
the  information  needed  if  it  is  too  difficult 
to  locate.  Whether  designing  a  local  infor¬ 
mation  system  or  a  DoD-wide  informa¬ 
tion  portal,  the  fundamental  principles  of 
successful  Web-based  KM  systems  are  the 
same: 

1 .  Minimize  bells  and  whistles  and  maxi¬ 
mize  quick  access  and  simplicity  of 
operation. 

2.  As  the  level  of  site  complexity  and 
menus  rise,  the  level  of  user  utility 
diminishes. 

3.  Needed  information  should  be  no 
more  than  three-to-five  mouse  clicks 
to  user  acquisition,  with  three  being 
the  technical  objective. 

4.  Keep  the  site  menu  structure  as  shal¬ 
low  as  possible. 

5.  A  graphics-based  system  is  normally 
more  user-friendly  than  a  text-based 
system,  and  a  duplex  system  (a  system 
that  uses  both  text  and  graphic-based 
methods  to  retrieve  information)  can 
be  more  effective  than  a  graphics- 
based  system  alone. 

6.  Focus  system  design  and  technical 
architecture  on  speed,  easy  access,  and 
simplicity. 

7.  Accurate  and  intuitive  titling  of  data 
descriptors,  menus,  titles,  or  entry 
points  is  extremely  important. 

8.  Organize  information  by  process, 
activities,  functions,  and  organization 
(as  appropriate  for  your  needs). 
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Figure  1 :  The  DTRA  Acquisition  Process 


ToolBook  Structure 

In  this  particular  architectural  design,  the 
ToolBook  was  structured  to  follow  the 
DTRA  acquisition  process  from  program 
start  to  program  closeout  activities  (see 
Figure  1).  The  site’s  home  page  is  divided 
into  three  broad  phases:  Early  Prepa¬ 
ration,  Pre- Award  Activities,  and  Program 
Execution.  Early  Preparation  contains  the 
initial  activities  required  for  up-front 
acquisition  project  planning  and  organiza¬ 
tion.  The  Pre-Award  Activities  section 
includes  all  follow-on  acquisition  and  con¬ 
tractual  efforts  to  get  the  acquisition 
awarded  and  on-contract.  The  Program 
Execution  section  contains  information 
on  the  post-award  phase,  which  includes 
program  management  (PM)  and  oversight 
activities  required  to  administer  and  exe¬ 
cute  a  successful  program.  The  ToolBook 
home  page  graphic  portrayal  of  the 
DTRA  acquisition  process  is  organized 
into  24  activity  boxes  that  form  a  logical 
progression  of  the  work  activities  required 
to  get  an  acquisition  effort  on-contract 
and  executed.  There  is  also  one  box  enti¬ 
tled  General  PM  Preferences  that  contains 
broad-based  or  overarching  documents 
that  do  not  fit  into  any  one  activity  box 
category.  Although  the  ToolBook  is  pri¬ 
marily  a  graphics-based  acquisition  portal, 


it  is  actually  composed  of  a  duplex  archi¬ 
tecture  that  can  use  a  graphically-based 
methodology  to  search  and  retrieve  data 
or  a  text-based  library  view  that  can  quick¬ 
ly  locate  and  more  effectively  display  relat¬ 
ed  task  data.  The  choice  of  method  used 
is  based  on  the  user’s  preference. 
Providing  the  user  with  information  dis¬ 
play  options  increases  a  program’s  utility. 

There  is  only  one  main  sub-level  menu 
for  each  activity  box  in  the  main 


ToolBook  that  houses  the  majority  of 
documents,  making  users  no  more  than 
three  mouse  clicks  away  from  most  of  the 
information  they  need  (see  Figure  2). 
There  is  also  one  third-level  menu  for 
unique  enterprise-level  documents.  Within 
each  activity  box  in  die  second-level  menu 
are  separate  icons  for  the  following  six 
KM  information  areas:  Tools  and 
Examples,  Policy  Documents,  Issuances 
(which  contains  guides,  manuals,  hand- 


Figure  2:  The  Sub-Level  Menu  for  When  A  User  Need  Is  Identified 
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books,  etc.),  Training,  Ask  an  Expert,  and 
Enterprise-Unique  Documents. 

The  ToolBook  uses  a  progressive  informa¬ 
tion  approach  to  information  classification 
and  management.  For  example,  if  a  pro¬ 
ject  manager  is  unfamiliar  with  award  fee 
contracts  and  requires  information  on 
how  to  write  an  award  fee  plan,  the 
ToolBook  offers  a  progressive  level  of 
knowledge  to  help  the  user  get  the  job 
done.  First,  die  user  would  select  the 
Award  Fee  activity  box.  When  the  second- 
level  menu  appears,  the  five  main  icons 
provide  a  graduated  pyramid  level  of 
information.  The  Training  icon  would 
provide  die  user  with  basic  information 
on  the  concepts,  responsibilities,  and 
requirements  of  award  fee  contracts  and 
issues.  If  more  detailed  information  is 
required,  the  Issuances  icon — which 
includes  an  array  of  in-depdi  guides,  man¬ 
uals,  handbooks,  standard  operating  pro¬ 
cedures,  and  standard  operating  instruc¬ 
tions — will  provide  a  multitude  of 
detailed  information  on  the  subject.  Once 
training  and/or  detailed  information  is 
accessed  on  the  subject,  the  user  can  select 
die  Tools  and  Examples  icon  that  pro¬ 
vides  the  actual  examples,  checklists,  and 
templates  needed  to  help  complete  die 
task  at  hand.  The  Policy  icon  provides  any 
relevant  policy  memoranda  on  die  subject. 

As  an  avenue  of  last  resort,  the 
ToolBook  also  features  a  sophisticated  Ask 
an  Expert  capability  diat  permits  users  to 
send  acquisition-related  questions  to 
agency  experts  on  die  subject.  For  enter¬ 
prise-unique  processes,  procedures,  and 
instructions,  users  can  also  access  dieir  own 
enterprise’s  menu  of  key  documents  man¬ 
aged  by  each  enterprise.  The  ToolBook  also 
includes  a  directory  of  Internet  links  to 
nearly  all  key  agency  and  DoD  acquisition 
references  as  well  as  to  Web  pages  explain¬ 
ing  how  to  perform  subsidiary  tasks  (such 
as  die  completion  of  travel  forms  required 
for  the  Defense  Travel  System).  The 
ToolBook  also  supports  a  document  search 
function  and  a  library  view  capability  that 
can  simultaneously  display  documents  by 
each  category  (for  a  particular  activity  box) 
for  all  documents. 

Pros  and  Cons 

The  ToolBook  system  incorporates  nu¬ 
merous  advantages  to  both  users  and  sys¬ 
tem  administrators.  It  is  designed  for  a  low 
user  investment  in  time  and  training,  and 
also  for  low  administrative  burden.  The 
Microsoft  SharePoint  2007  architecture  is 
easy  for  system  administrators  to  manage, 
and  offers  powerful  features.  With  some 
customization  during  the  development 
process,  it  is  easy  for  content  managers  to 
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load  document  files  and  links  into  the 
ToolBook  library.  Adding,  deleting,  and 
modifying  documents  is  a  simple  process. 
Formal  user  training  classes  are  not 
required;  a  narrated  internal  virtual  tour 
movie  provides  users  with  an  overview  of 
die  entire  ToolBook  site.  After  an  initial 
promotional  campaign  and  scheduled  sys¬ 
tem  demonstrations,  the  site  is  ready  for 
full  operation  upon  release.  Although  it 
would  be  recommended  to  split  the 
responsibilities  of  system  administra¬ 
tor/developer  and  content  manager,  the 
DTRA  ToolBook  development,  adminis¬ 
trator,  and  content  management  tasks  are 
assigned  to  one  individual.  Furthermore, 
die  Ask  an  Expert  function  is  also  man¬ 
aged  by  the  system  administrator.  Finally, 
system  capability  is  easily  expandable  and 
die  initial  system  development  cost  is  low 
using  proven  software  such  as  Microsoft 
SharePoint,  Adobe  Flash,  and  Microsoft 
.NET  programming. 

Like  similar  systems,  there  are  some 
drawbacks.  System  data  content  must  be 
reviewed  for  validity  and  utility,  and  to 
ensure  that  it  includes  updated  informa¬ 
tion  at  least  every  three  to  six  months. 
This  could  require  the  individual  reassess¬ 
ment  of  hundreds  of  documents  quarter¬ 
ly  if  they  are  not  linked  to  golden  sources3. 
Long-term  system  maintenance  support 
will  be  required  from  either  internal  IT 
resources  or  an  outside  contractor. 
Periodic  changes  may  be  needed  that 
require  programming  modifications,  and 
future  updates  and  development  efforts 
will  require  IT  support.  Additionally,  some 
users  may  have  difficulty  locating  needed 
information  if  they  do  not  have  a  mini¬ 
mum  understanding  of  the  Federal 
Acquisition  Regulations  process.  The  con¬ 
tent  manager  may  have  to  place  a  docu¬ 
ment  in  more  than  one  activity  box,  result¬ 
ing  in  some  redundancy,  but  reducing 
search  time. 

Conclusion 

The  Uttle  A  principles  of  acquisition  KM 
appear  to  apply  to  Big  A  acquisition  por¬ 
tals.  Both  have  a  specific  set  of  users  that 
demand  similar  attributes  regarding  system 
operability:  Operational  simplicity,  swift 
data  location  and  extraction,  and  a  logical 
taxonomy  and  data  organization  scheme  to 
find  and  manipulate  acquisition  data.  The 
fusion  of  the  key  principles  of  KM  and 
specific  organizational  processes  represent 
a  merger  of  KM,  process  management, 
and  operational  simplicity — the  founda¬ 
tional  triad  of  successful  user  information 
systems.  The  DTRA  Acquisition  Tool¬ 
Book  has  effectively  managed  to  integrate 
the  positive  elements  of  portal  and 


process  development  to  the  benefit  of  its 
acquisition  workforce.  ♦ 

Notes 

1 .  Any  military  or  civilian  government 
agency  interested  in  developing  their 
own  Acquisition  ToolBook  can  arrange 
a  ToolBook  briefing  and  demonstration 
at  die  DTRA  by  contacting  Dr.  Avery. 

2.  For  those  not  in  die  acquisition  busi¬ 
ness,  Big  A  addresses  issues  such  as 
requirements  generation,  program  and 
budgeting,  sustainment,  development, 
production,  strategic  planning,  major 
milestone  processes,  and  the  procure¬ 
ment  process.  Uttle  A  is  procurement 
or  acquisition  in  its  narrowest  sense. 

3.  Golden  sources  are  those  that  are  auto¬ 
matically  updated  by  the  source  orga¬ 
nization  that  either  created  or  is 
responsible  for  updating  die  document 
(as  required). 
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Sandy  Schwalb 

Defense  Technical  Information  Center 

We  all  seem  to  be  doing  more  with  less  these  days.  Take  a  few  minutes  to  learn  about  what  is  available  at  the  Defense 
Technical  Information  Center  (DTIC),  an  organisation  that  can  save  you  time  and  money.  The  DTIC  offers  a  world  of 
information  at  your fingertips,  whether  you  use  a  desktop  or  a  laptop.  The  DTIC  has  information  that  is  from  the  defense 
community,  about  the  defense  community,  and for  the  dfense  community. 


The  DTIC  is  currently  the  largest  cen¬ 
tral  resource  available  for  DoD  and 
government-funded  scientific,  technical, 
engineering,  and  business-related  infor¬ 
mation1.  For  more  than  60  years,  the 
DTIC  has  provided  warfighters,  engineers, 
researchers,  scientists,  and  those  in  labora¬ 
tories  and  universities  access  to  more  than 
2  million  publications  covering  250  sub¬ 
ject  areas.  Approximately  100  Web  sites 
are  designed,  hosted,  and  maintained  by 
die  DTIC  for  other  DoD  agencies  and 
programs.  The  DTIC  also  manages  10 
Information  Analysis  Centers  (IACs), 
which  locate,  analyze,  and  provide  scien¬ 
tific  and  technical  (S&T)  information  in 
specialized  subject  areas. 

The  DTIC — a  DoD  field  activity  with¬ 
in  die  office  of  die  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and 
Logistics  that  reports  to  the  Defense 
Research  Engineering  Director — is  one  of 
several  organizations  whose  work  reaches 
across  all  segments  of  the  DoD. 

Why  Use  the  DTIC?  Why  Not? 

The  Information  Age  is  also  the  era  of 
reduced  budgets  and  fewer  personnel, 
where  doing  more  widi  less  is  an  overused 
(but  true)  expression.  This  is  certainly  a 
reality  in  the  federal  government.  The 
DoD  Scientific  and  Technical  Infor¬ 
mation  Program  Directive  3200.12  calls 
for  the  need  to  reduce  unnecessary  S&T 
expenditures.  This  is  where  DTIC  comes 
in.  Using  die  DTIC’s  resources  can  help: 

•  Leverage  the  multi-billion  dollar 
investment  in  DoD  research  and  engi¬ 
neering. 

•  Prevent  unnecessary  or  redundant 
research. 

•  Get  DoD  S&T  information  into  the 
hands  of  the  right  people  in  the 
defense  community. 

•  DoD  researchers  and  decision  makers 
turn  DTIC  data  into  value-added 
knowledge  for  ongoing  and  new 
efforts. 

•  Offer  up-to-date  electronic  informa¬ 
tion  to  the  defense  community  using 

push  technology. 


•  Provide  information  support  to  the 
federal  and  contractor  communities. 

•  Build  on  prior  knowledge. 

Why  reinvent  the  wheel ?  Sharing  re¬ 
sources  can  be  a  boon  to  engineers, 
researchers,  students,  and  anyone  who 
needs  information  to  produce  new  and 
better  research  that  can  lead  to  new  and 
better  technologies.  We  want  to  ensure 
that  our  collection  is  as  complete  as  possi¬ 
ble,  allowing  researchers  to  find  informa- 

“Why  reinvent  the 
wheel?  Sharing  resources 
can  be  a  boon  to 
engineers,  researchers, 
students,  and  anyone 
who  needs  information 
to  produce  new  and 
better  research  that  can 
lead  to  new  and  better 
technologies.  ” 

tion,  determine  if  money  has  been  spent 
on  that  kind  of  effort  in  the  past,  and 
learn  about  those  project  outcomes. 

One  Site,  One  Source 

The  December  2007  CROSSTALK  Web 
Sites  section  highlighted  one  of  our  data¬ 
bases: 

The  Public  Scientific  and  Technical 
Information  Network  (STINET)  is 
available  to  the  general  public,  free 
of  charge.  It  provides  access  to  cita¬ 
tions  of  unclassified  unlimited  doc¬ 
uments  that  have  been  entered  into 
the  DTIC’s  Technical  Reports 
Collection,  as  well  as  die  electronic 


full-text  of  many  of  these  docu¬ 
ments.  Public  STINET  also  pro¬ 
vides  access  to  die  Air  University 
Ubrary  Index  to  military  periodi¬ 
cals,  staff  College  Automated 
Military  Periodical  Index,  DoD 
Index  to  Specifications  and  Stan¬ 
dards,  and  Research  and  Develop¬ 
ment  Descriptive  summaries.  [1] 

With  the  advent  of  the  DTIC’s 
redesigned  Web  site  and  search  systems 
<www.dtic.mil/dtic/search/tr/>,  die  ac¬ 
ronym  STINET  is  no  longer  used  for  our 
public  site.  The  good  news  is  that  all  of 
the  information  in  our  collection  (and 
then  some)  is  still  offered. 

As  a  result  of  the  rapid  growth  of 
DTIC  capabilities  and  inclusion  of  new 
information  audiences,  the  DTIC’s  admin¬ 
istrator,  R.  Paul  Ryan,  directed  die  staff  (in 
2007)  to  create  a  more  structured  (sim¬ 
pler)  information  delivery  approach.  The 
goal  was  met  in  mid-2008  when  the  DTIC 
launched  a  new  user  interface,  DTIC 
Online  <www.dtic.mil>,  which  offers  a 
one-site,  one-source  location  for  DoD 
S&T  information. 

DTIC  Online  provides  free  access  to 
citations  of  public  release  reports  that 
describe  the  progress  or  results  of 
research  efforts  and  other  S&T  informa¬ 
tion  held  by  the  DTIC.  Many  of  these 
documents  are  available  in  full-text  and 
can  be  downloaded.  A  key  feature  of  the 
new  site  is  the  ability  to  search  more  data¬ 
bases  using  DTIC’s  MultiSearch  En¬ 
hanced  <http: //multisearch. dtic.mil>. 
This  tool,  a  portal  to  the  deep-Web  for  gov¬ 
ernment  S&T  information,  searches  con¬ 
tent  below  the  surface  Web  for  information 
not  accessible  through  commercial  and 
government  search  engines.  This  search 
feature  assists  the  DoD  community  in 
accessing  S&T  information  over  a  wide 
range  of  DoD  and  commercial  sources. 
DTIC  customers  can  now  search  an  esti¬ 
mated  3.5  million  documents,  from  more 
than  50  national  and  international  sources. 

New  to  this  public  DTIC  Web  site  is 
the  Interest  Areas  section,  which  provides 
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access  to  a  broad  range  of  contacts,  asso¬ 
ciations,  blogs,  conferences,  and  research 
institutions  appropriate  to  S&T  research 
communities.  Each  interest  area  has  a 
DTIC  staff  member  who  manages  the 
specific  page  <www.dtic.mil/ dtic/ com 
munities/>.  Feedback  and  comments  are 
welcome:  <ref@dtic.mil>. 

The  DTIC  is  currently  assessing  various 
avenues  as  we  look  to  revamp  our  access- 
controlled  site,  available  to  our  registered 
customers  (see  die  registration  section). 

What  Is  Available? 

You  can  find  documents  on  topics  ranging 
from  acquisitions  to  zeta  functions: 

•  Public  Technical  Reports.  This 
database  contains  more  dran  2  million 
reports  in  print  and  non-print  formats 
(software,  datafiles,  databases,  and 
video  recordings)  conveying  the  results 
of  defense-sponsored  research,  devel¬ 
opment,  test,  and  evaluation  efforts.  It 
includes  journal  articles,  DoD-spon- 
sored  patent  applications,  studies, 
analyses,  open-source  literature  from 
foreign  countries,  conference  proceed¬ 
ings,  and  theses.  Between  30,000  and 
35,000  new  documents  are  added 
annually. 

•  Research  Summaries.  This  database 
contains  descriptions  of  DoD  re¬ 
search  in  progress,  providing  informa¬ 
tion  on  technical  content,  responsible 
individuals  and  organizations,  princi¬ 
pal  investigators,  and  funding  sources2. 
The  collection  consists  of  approxi¬ 
mately  309,000  active  and  inactive 
summaries  from  1965  to  the  present. 

•  Independent  Research  and  Devel¬ 
opment  (IR&D).  This  database  con¬ 
tains  more  than  173,000  descriptions 
(dating  back  to  the  mid-’70s)  of  R&D 
projects  initiated  and  conducted  by 
defense  contractors  independent  of 
DoD  control  and  without  direct  DoD 
funding.  On  average,  in  excess  of  2 
billion  dollars  worth  of  IR&D  pro¬ 
jects  are  annually  submitted  to  the 
DTIC.  The  database  includes  basic 
and  applied  research  and  technology 
development  efforts  as  well  as  systems 
and  concept  formulation  studies. 
Defense  contractors  and  potential 
contractors  are  encouraged  to  submit 
project  descriptions  to  the  IR&D 
database.  Accessible  only  to  DoD  and 
select  U.S.  government  organizations, 
the  proprietary  IR&D  information  is 
used  to  identify  contractors  with 
expertise  in  areas  of  interest  to  the 
DoD  and  to  avoid  DoD  duplication 
of  industry  research  and  development 
efforts. 


•  Technical  Reports  Automated  In¬ 
formation  List.  A  free,  publicly  avail¬ 
able  electronic  mailing  list  that,  every 
two  weeks,  disseminates  citations  to 
die  DTIC’s  recently  added  unclassi¬ 
fied,  unlimited  technical  reports. 

A  Leader  in  Utilizing  the  Web 

As  a  center  of  excellence  for  information 
storage  and  retrieval,  the  DTIC  has  been 
able  to  advise  DoD  components  concern¬ 
ing  policy,  law,  best  practices,  and  security 
strategies  that  relate  to  die  transmission 
and  use  of  all  types  of  information.  The 
DTIC  offers  full-service  Web  support,  an 
established  Web  architecture,  and  cus¬ 
tomer  liaisons.  The  DTIC  hosts  more 
dian  100  Web  sites,  such  as  the  Joint 

“As  a  center  of 
excellence  ...  the  DTIC 
has  been  able  to  advise 
DoD  components 
concerning  policy,  law, 
best  practices,  and 
security  strategies  that 
relate  to  the 
transmission  and 
use  of  all  types 
of  information.  ” 

Chiefs  of  Staff,  die  Defense  Prisoner  of 
War/Missing  in  Action  Office,  and  the 
Federal  Voting  Assistance  Program. 

To  Distribute  or  Not  to 
Distribute 

While  there  is  much  publicly  accessible 
material  in  die  DTIC  collection  (almost 
one-half  of  die  DoD’s  technical  reports 
are  publicly  available  the  day  they  are  pub¬ 
lished),  some  information  has  security 
classifications.  The  DoD’s  S&T  informa¬ 
tion  is  always  categorized  (or  marked,  the 
term  used  in  the  defense  community)  by 
die  office  that  originates  die  document. 
This  marking  determines  how  and  with 
whom  the  information  can  be  shared. 
Some  information  is  marked  to  protect 
national  security.  DTIC  databases  contain 
such  classified  information  that  may  be 
marked  confidential  or  secret. 

DTIC  databases  also  contain  informa¬ 


tion  drat,  although  not  classified,  is  still 
sensitive  for  various  reasons.  These  docu¬ 
ments  are  marked  to  show  why  the  infor¬ 
mation  is  sensitive  and  to  whom  the  doc¬ 
ument  can  be  distributed.  Such  docu¬ 
ments  are  referred  to  as  unclassified,  limited. 
Information  that  is  neither  classified  nor 
limited  (but  can  be  released  to  the  public) 
is  referred  to  as  unclassified,  unlimited.  The 
DTIC’s  collection  is  comprised  of  51  per¬ 
cent  unclassified,  unlimited;  40  percent 
unclassified,  limited;  and  9  percent  classi¬ 
fied  information. 

Registration  Is  the  Key 

The  first  step  in  ensuring  that  you  can  get 
the  information  you  need  from  us  is  by 
registering  for  services  at  <www.dtic.mil/ 
dtic/ registrations*.  Access  to  DoD  classi¬ 
fied  and  unclassified,  limited  information 
is  controlled  through  this  registration 
process.  A  variety  of  secure  DoD  Web 
sites  can  be  accessed  by  audiorized  users. 
While  our  primary  customers  are  those 
who  have  a  legitimate  business  relation¬ 
ship  with  the  DoD  and  largely  include  the 
military  and  defense  contractors,  there  are 
other  categories  of  customers. 

Who  uses  DTIC  information?  Here  is 
a  snapshot  of  our  registered  customers: 

•  Acquisition  instructors. 

•  Active  duty  military. 

•  Congressional  staff. 

•  Directors  of  corporate  relations. 

•  DoD  contractors. 

•  Faculty  at  military  schools. 

•  Historians. 

•  Logistics  management  specialists. 

•  Quality  assurance  specialists. 

•  Small  business  owners. 

•  Security  managers. 

•  Software  engineers  and  developers. 

A  Wealth  of  Information 

The  DTIC  has  a  wide  range  of  informa¬ 
tion  products  relating  to  S&T  planning, 
budget,  financial,  R&D  descriptions,  man¬ 
agement,  test  and  evaluation,  research 
results,  training,  law,  command  histories, 
conference  proceedings,  DoD  Directives 
and  Instructions,  foreign  documents  and 
translations,  journal  articles,  management 
summaries,  security  classification  guides, 
technical  reports,  and  summaries  of  works 
in  progress. 

Why  does  the  DTIC  get  this  informa¬ 
tion?  It  is  required  by  DoD  Instruction 
3200.14,  which  mandates  that  DoD 
research,  including  research  done  in-house 
and / or  by  contractors  and  grantees,  should 
be  part  of  die  DTIC  collection.  In  other 
words,  if  drere  is  great  technology  in  the 
DoD,  the  DTIC  should  have  that  informa¬ 
tion  for  others  to  use  and  build  upon.  The 
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material  comes  from  many  sources: 

•  DoD  organizations  (civilian  and  mili¬ 
tary)  and  DoD  contractors. 

•  U.S.  government  organizations  and 
their  contractors. 

•  Non-profit  organizations  working  on 
DoD  scientific,  research,  and  engineer¬ 
ing  activities. 

•  Academia. 

•  Foreign  governments. 

Specialized  Information 
Solutions 

Another  facet  of  DTIC  activities  is  the 
management  and  funding  of  10  contrac¬ 
tor-operated  joint  service-oriented  IACs 
<http://iac.dtic.mil>.  Chartered  by  the 
DoD,  IACs  locate  and  analyze  S&T  infor¬ 
mation  for  customers  in  specific  subject 
areas.  Staffed  by  experienced  technical 
area  scientists,  engineers,  and  information 
specialists,  the  IACs  have  comprehensive 
knowledge  bases  which  include  historical, 
technical,  scientific,  and  other  data  collect¬ 
ed  on  a  worldwide  basis.  Many  of  the 
IACs’  products  and  services  are  free  and 
include  announcements  of  pertinent 
reports  in  the  particular  IAC’s  field  of 
interest,  authoritative  bibliographic  search 
reports,  the  latest  scientific  and  engineer¬ 
ing  information  on  specific  technical  sub¬ 
jects,  consultation  with  or  referral  to 
world-recognized  technical  experts,  and 
die  status  of  current  technologies.  The  10 
DTIC-managed  IACs  (as  of  September 
2008)  are: 

•  AMMTIAC:  Advanced  Materials, 
Manufacturing,  and  Testing. 

•  CBRNIAC:  Chemical,  Biological, 
Radiological,  and  Nuclear  Defense. 

•  CPIAC:  Chemical  Propulsion. 

•  DACS:  Data  and  Analysis  Center  for 
Software. 

•  IATAC:  Information  Assurance  Tech¬ 
nology. 

•  MSIAC:  Modeling  and  Simulation. 

•  RIAC:  Reliability. 

•  SENSIAC:  Military  Sensing. 

•  SURVIAC:  Survivability/Vulnerability. 

•  WSTIAC:  Weapons  Systems  Technol¬ 
ogy- 

Technical  Area  Tasks  are  fee-based 
and  more  extensive  than  basic  IAC  prod¬ 
ucts  and  services.  They  vary  from  a  frac¬ 
tion  of  a  staff  year  to  several  staff  years 
and  can  cost  from  a  few  diousand  to  sev¬ 
eral  million  dollars.  These  tasks  can  be 
ordered  by  any  DoD  component  and  on  a 
case-by-case  basis  by  odier  federal  organi¬ 
zations. 

Annual  Conference 

The  DTIC’s  annual  conference  is  held 


each  spring  in  the  Washington,  D.C.  area. 
Attendees  typically  include  scientists,  engi¬ 
neers,  and  professionals  in  the  technology 
research,  development,  information  sci¬ 
ence,  and  acquisition  communities  repre¬ 
senting  die  DoD,  odier  federal  agencies, 
and  contractors.  The  diree-day  conference 
features  speakers  from  government,  pri¬ 
vate  industry,  and  the  DTIC  who  address 
evolving  information  technologies  and 
sources.  A  technology  expo  is  held  where 
exhibitors  tout  the  latest  in  scientific, 
research,  and  engineering  fields.  The  2009 
conference  is  scheduled  for  April  6-8  in 
Alexandria,  Virginia.  For  more  informa¬ 
tion  go  to:  <www.dtic.mil/dtic/announce 
ments/conference.html>. 

The  DTIC  Review 

This  quarterly  CD  publication  provides 
die  full-text  of  selected  technical  reports 
and  a  bibliography  of  odier  references  of 
interest  in  a  single  package.  Each  volume 
provides  a  sampling  of  documents  from 
die  DTIC  collection  on  a  specific  topic  of 
current  interest.  Titles  in  2008  included: 
Non -Lethal  Weapons,  Intelligent  Autonomous 
Vehicles,  and  Human,  Social,  Cultural,  and 
Behavior  Modeling. 

Training 

Free  training  in  searching  the  DTIC’s 
databases  and  handling  of  DoD  technical 
information  is  offered  to  all  DTIC-regis- 
tered  customers  at  our  headquarters 
(McNamara  Headquarters  Complex,  Fort 
Belvoir,  VA)  and  four  regional  offices 
(Boston,  MA;  Dayton,  OH;  Albuquerque, 
NM;  and  Los  Angeles,  CA).  Customized 
and  on-site  courses  can  be  provided,  with 
travel  expenses  paid  by  the  hosting  orga¬ 
nization.  For  more  information,  go  to: 
<www.dtic.mil/ dtic/customer/ training/ 
index. html>. 

No  More  Dead  Pages 

The  DTIC  is  committed  to  maintaining 
die  permanent  availability  of  the  informa¬ 
tion  in  its  collection.  Here’s  a  real-life 
example:  You  are  working  against  a  dead¬ 
line  and  find  a  Web  site  that  has  a  needed 
resource;  you  click  on  die  link  and  are 
taken  to  a  dead  page  with  the  dreaded 
Error  404  message.  A  solution  to  diis 
problem  is  die  DTIC’s  Handle  Service 
<www.dtic.mil/ dtic/ stresources/ dtic 
SearchTools/handleservice.html>.  What 
exactly  is  a  handle?  It  is  a  permanent  name 
for  a  digital  object  (publication,  article, 
research  paper,  and  so  forth)  regardless  of 
where  and  how  the  object  is  stored.  In 
other  words,  it  provides  long-term  access 
to  a  digital  resource.  Handles,  added  to 
public  information  in  DTIC  collections, 


link  directly  to  full-text  PDF  documents. 

The  Power  of  Information 

The  DTIC  gets  information  from  and  for 
the  defense  community — about  defense 
issues  and  beyond.  Having  a  full  range  of 
S&T  and  R&D  within  our  collection 
ensures  that  technological  innovations  are 
linked  to  defense  development  and  acqui¬ 
sitions  efforts.  New  research  efforts  can 
begin  with  the  highest  level  of  informa¬ 
tion  available.  This,  in  turn,  maximizes  the 
use  of  DoD  project  dollars.  ♦ 
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Notes 

1 .  The  DTIC  has  been  successful  in  pro¬ 
viding  information  to  the  defense 
community  by  knowing  how  to  pre¬ 
sent  information.  The  DTIC  is  a  vital 
link  in  the  partnerships  among  war¬ 
fighters,  the  acquisition  community, 
technology  providers,  and  information 
professionals.  No  matter  what  the 
medium,  content,  or  dissemination 
method,  the  DTIC’s  job  has  always 
been  to  help  people  use  information  as 
efficiently  as  possible  to  strengthen  the 
nation’s  defense.  Visit  us  at  <www. 
dtic.mil>. 

2.  Only  available  to  registered  customers, 
access  to  dais  collection  is  based  on 
individual  access  restrictions. 
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Project  Duration  Forecasting:  Comparing  Earned  Value 
Management  Methods  to  Earned  Schedule 

Walt  Lipke 

Oklahoma  City  Chapter  of  the  Project  Management  Institute 

Earned  I  ralue  Management  (El  M)  methods  for forecasting  project  duration  have  been  taught  in  training  courses  and  used 
by  project  managers  for four  decades.  These  El  M  methods  are  generally  considered  to  be  accepted  practice,  yet  they  have  not 
been  well- studied  and  researched  as  to  their  predictive  capability.  Using  real project  data,  this  article  examines  and  compares 
the  duration  forecasts  from  four  El  M  methods  to  the  Earned  Schedule  (ES)  prediction  technique. 


The  concept  of  ES  was  introduced  in 
the  spring  of  2003,  demonstrating  the 
possibility  of  describing  schedule  perfor¬ 
mance  in  units  of  time  [1],  ES  facilitates 
time-based  analysis  of  the  schedule, 
employing  uniquely  the  EVM  measures  of 
cost.  One  year  subsequent  to  the  publica¬ 
tion  of  ES,  the  concept  was  extended  to 
include  project  duration  forecasting.  The 
article  “Further  Developments  in  Earned 
Schedule”  [2]  put  forth  two  equations  for 
forecasting  die  final  duration  for  a  project, 
one  of  which  is  used  in  this  study. 

From  2004-2007,  two  independent 
papers  were  published  investigating  the 
capability  of  the  ES  forecasting  method. 
One  paper  written  by  Lew  Flecht 
describes,  positively,  the  usefulness  of  ES 
in  a  case  study  of  a  single  U.S.  Navy  pro¬ 
ject  [3],  The  second  is  a  comprehensive 
examination  of  the  capability  of  ES.  The 
research  team  of  Vanhoucke  and  Vande- 
voorde  applied  a  simulation  method  for 
assessing  the  performance  of  two  EVM- 
based  methods  and  ES  in  forecasting  pro¬ 
ject  duration  [4],  A  portion  of  the 
Vanhoucke  and  Vandevoorde  paper  has 
been  updated  and  published  in  the  Winter 
2007-2008  issue  of  The  Measurable  News 
[5]1.  The  conclusion  from  both  is  that: 
“The  results  ...  confirm  ...  that  the  Earned 
Schedule  method  outperforms,  on  aver¬ 
age,  the  other  forecasting  methods.” 

Although  the  results  of  die  research 
performed  by  Vanhoucke  and  Vande¬ 
voorde  are  well  regarded,  there  remains 
die  question  of  whedier  the  simulation 
technique  is  truly  representative  of  real 
project  circumstances.  Likewise,  the  case 
study  testimonial,  while  strongly  support¬ 
ive  of  the  use  of  ES  indicators  and  fore¬ 
casting,  is  inconclusive  in  broadly  validat¬ 
ing  the  concept.  Beyond  die  recognized 
shortcomings  of  the  aforementioned 
studies,  it  has  recently  been  recognized 
that  four  frequently  used  EVM-based 
methods  of  duration  forecasting  have  not 
been  compared  to  ES.  This  research  is 
focused  to  overcome  the  identified  gaps. 
Real  data  from  16  projects  is  used  to  ana¬ 


lyze  the  respective  forecasting  capabilities 
of  the  overlooked  EVM  methods  along 
with  ES. 

This  article  begins  by  defining  the  per¬ 
tinent  elements  of  die  EVM  and  ES 
methods.  Building  on  this  foundation,  the 
forecasting  equations  are  presented.  Next, 
the  hypothesis  of  the  analysis  is  described. 
Then  die  computations  needed  to  per¬ 
form  the  analysis  and  evaluation  are  out¬ 
lined.  The  project  data  is  then  character¬ 
ized  and  results  from  the  computations 
and  analysis  are  discussed.  Finally,  conclu¬ 
sions  are  drawn. 

EVM  Duration  Forecasting 

An  understanding  of  EVM  and  its  termi¬ 
nology  is  assumed  in  this  article.  For  con¬ 
venience,  die  EVM  terminology  used  to 
portray  project  status  and  forecast  final 
duration  is  defined  in  the  following: 

•  Planned  Value  (PV). 

•  Earned  Value  (EV) . 

•  Budget  at  Completion  (BAC),  which  is 
die  planned  cost  of  the  project. 

•  Performance  Measurement  Baseline 
(PMB),  which  is  the  cumulative  PV 
over  time. 

•  Independent  Estimate  at  Completion 
(IEAC(t)),  which  is  the  forecast  final 
duration. 

Four  EVM  duration  forecasting  tech¬ 
niques  have  been  commonly  applied  over 
die  last  40  years  to  predict  project  com¬ 
pletion  dates.  These  methods  have  the  fol¬ 
lowing  basic  form: 

Duration  Forecast  =  Elapsed  Time  + 
Forecast  for  Work  Remaining 
IEAC(t)  =  AT  +  (BAC  -  EV)  /  Work  Rate 

where 

AT  =  Actual  Time  (the  duration  elapsed  to 
the  time  at  which  PV  and  EV  are  mea¬ 
sured) 

BAC  -  EV  is  commonly  termed  the  work 
remaining 

Work  Rate  is  a  factor  which  converts  the 
work  remaining  to  time,  the  duration  fore¬ 
cast  for  the  remaining  work 


The  four  Work  Rates  commonly  applied 
are: 

1)  Average  Planned  Value:  PVav  = 
PVcum/n 

2)  Average  Earned  Value:  EVav  = 

EVcum/n 

3)  Current  Period  Planned  Value:  PVIp 

4)  Current  Period  Earned  Value:  EVIp 

where 

PVcum  =  Cumulative  value  of  PV 
EVcum  =  Cumulative  value  of  EV 
n  =  Total  number  of  periodic  time  incre¬ 
ments  of  project  execution  within  AT 

The  EVM  forecasts  of  final  duration, 
IEAC(t),  are  associated  widi  die  Work 
Rate  employed  and  identified  in  the 
remainder  of  this  article  as  follows: 

1)  PVav:  IEAC(t)PVav 

2)  EVav:  IEAC(t)EVav 

3)  PVIp:  IEAC(t)PVIp 

4)  EVIp:  IEAC(t)EVIp 

ES  Duration  Forecasting 

A  recent  extension  to  EVM,  ES,  has 
emerged  to  provide  reliable,  useful  sched¬ 
ule  performance  management  informa¬ 
tion.  In  brief,  the  mediod  yields  time- 
based  indicators,  unlike  die  cost-based 
indicators  for  schedule  performance 
offered  by  EVM. 

Figure  1  is  an  illustration  for  under¬ 
standing  the  concept.  The  ES  measure 
identifies  when  the  amount  of  EV 
accrued  should  have  occurred.  As  depict¬ 
ed  by  the  diagram,  this  is  the  point  on  the 
PMB  where  PV  equals  the  EV  accrued. 
The  vertical  line  from  the  point  on  the 
PMB  to  the  time  axis  determines  the 
earned  portion  of  the  schedule.  The  dura¬ 
tion  from  the  beginning  of  the  project  to 
the  intersection  of  die  time  axis  is  the 
amount  of  ES. 

With  ES  and  AT  defined,  the  schedule 
performance  efficiency  is  formulated  as 
depicted  in  Figure  1,  Schedule  Perfor- 
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Time-based  schedule  performance  efficiency:  SPI(t)  =  ES  /  AT 


Figure  1 :  The  Earned  Schedule  Concept 


mance  Index  (time)  [SPI(t)]  =  ES/AT. 
From  EVM,  final  cost  may  be  forecast 
from  the  formula,  IEAC  =  BAC/Cost 
Performance  Index  (CPI);  CPI  =  EV/AC, 
where  AC  is  the  actual  cost.  In  an  analo¬ 
gous  manner,  final  duration  is  forecast 
from  IEAC(t)es  =  PD/SPI(t),  where  PD 
is  the  planned  duration  for  the  project  and 
IEAC(t)es  is  the  ES  forecast  of  final  dura¬ 
tion. 

Discussion  of  Forecasting 
Methods  and  Study 
Considerations 

The  objective  of  the  study  is  to  investigate 
and  understand  the  forecasting  capability 
of  the  five  methods,  four  from  EVM  and 
one  from  ES.  By  inspection,  it  can  be 
deduced  that  the  EVM  Work  Rates  have 
mathematical  failings  which  affect  their 
performance. 

When  the  project  executes  past  its 
planned  duration,  PVcum  is  equal  to  its 
maximum  value,  BAC,  and  is  invariant 
thereafter.  Thus,  the  PVav  Work  Rate 
becomes  PVav  =  BAC/m,  where  m  is  a 
number  larger  than  the  planned  number 
of  time  periods  for  the  project.  Obviously, 
as  m  becomes  larger,  PVav  is  decreasingly 
smaller,  thereby  causing  the  work  remain¬ 
ing  forecast  to  be  longer  than  its  planned 
time. 

The  situation  for  the  PVlp  Work  Rate 
is  more  severe.  After  the  planned  project 
duration  has  passed,  there  are  no  periodic 
values  of  PV,  thereby  making  the  compu¬ 
tation  of  IEAC(t)PVlp  indeterminate. 
These  observations  are  excluded  from  the 
study  because  it  may  be  that  IEAC(t)PVlp 
is  a  good  predictor  otherwise.  A  tenet  of 
the  study  is  to  provide  each  method  a  rea¬ 
sonable  opportunity  to  show  well,  despite 
the  known  limitations. 

The  two  Work  Rates,  EVav  and  EVlp, 
do  not  normally  have  indeterminate  calcu¬ 
lation  conditions.  There  is,  however,  one 
exception  of  when  a  period  elapses  with 
no  EV  accrued;  this  condition  may  occur 
for  smaller  projects  which  assess  their  sta¬ 
tus  weekly.  When  EVlp  is  equal  to  zero, 
IEAC(t)EVlp  cannot  be  calculated.  Just  as 
for  PVlp,  the  condition  is  accommodated 
in  the  study  so  as  to  not  discredit  the  over¬ 
all  forecasting  performance  of  EVlp. 
When  an  anomalous  instance  is  encoun¬ 


tered,  the  forecast  for  the  previous  valid 
observation  is  used. 

The  forecasting  from  ES  does  not 
experience  indeterminate  calculation  con¬ 
ditions.  A  common  positive  characteristic 
of  all  of  the  methods,  with  the  exception 
of  IEAC(t)PVlp,  is  that  they  converge  to 
the  actual  duration.  The  predictive  capa¬ 
bility  of  the  four  EVM-based  methods  in 
this  study  may  be  superior  to  the  two  test¬ 
ed  by  Vanhoucke  and  Vandevoorde  [4,5]; 
those  methods  did  not  necessarily  correct¬ 
ly  calculate  the  actual  outcome  duration  at 
completion. 

Study  Hypothesis  and 
Methodology 

The  conjecture  to  be  examined  in  the 
study  is  that  ES  provides  a  better  forecast¬ 
ing  method  of  final  project  duration  than 
the  four  methods  cited  previously  for 
EVM.  To  make  a  determination  concern¬ 
ing  this  conjecture,  the  extreme  case  will 
be  examined  and  tested.  The  test  is  con¬ 
structed  to  show  that  the  EVM  methods, 
as  an  aggregate,  produce  better  forecasts 
than  ES  does.  If  the  EVM  methods  are 
shown  to  be  superior  to  ES,  it  will  not  be 
known  which  one  of  the  EVM  methods  is 
better.  Thus,  if  this  is  the  determination, 


further  examination  will  be  necessary  to 
understand  the  circumstances  for  selecting 
the  appropriate  EVM  forecasting  method. 

The  hypothesis  from  the  preceding 
discussion  is  formally  defined  (by  [6])  as 
follows: 

Ho:  EVM  methods  produce  the  better 
forecast  of  final  project  duration 
Ha:  The  ES  method  produces  the  better 
forecast  of  final  project  duration 

where 

Ho:  The  null  hypothesis  (i.e.,  the  state¬ 
ment  to  be  validated)  and  Ha  is  the  alter¬ 
nate  hypothesis. 

The  statistical  testing  is  performed 
using  the  Sign  Test  applied  at  a  0.05  level 
of  significance  [7],  Assuming  each  of  the 
five  methods  has  an  equal  probability  of 
success,  the  probability  for  each  trial  is  0.8. 

Data  from  16  projects  is  used  for  gen¬ 
erating  the  forecasts  from  each  of  the 
methods.  These  forecasts  are  then  tested 
and  analyzed.  The  test  statistic  for  the 
hypothesis  test  is  computed  from  the 
number  of  times  the  EVM  methods  are 
observed  to  yield  the  better  forecast. 
Thus,  for  each  testing  condition  applied, 


Table  1 :  Schedule  Performance  for  Projects  in  the  Data  Set 


Schedule  Performance 

Project 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

Planned  Duration 

21m 

32m 

36m 

43m 

24m 

50m 

46m 

29m 

45m 

44  m 

17m 

50m 

81w 

25w 

25w 

19w 

Actual  Duration 

24m 

38m 

43m 

47m 

24m 

59m 

54m 

30m 

55m 

50m 

23m 

50m 

83w 

25w 

22w 

13w 

Legend:  m  =  months  w  =  weeks 
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Data  and  Data  Management 


Figure  2:  Final  Duration  Forecasting  Comparisons 


die  maximum  number  of  successes  for  the 
EVM  methods  is  16.  When  the  EVM 
methods  successes  are  fewer  than  10,  the 
test  statistic  has  a  value  in  the  critical  region 
(<  0.05).  A  value  in  the  critical  region  indi¬ 
cates  that  there  is  enough  evidence  to 
reject  the  null  hypothesis.  In  clearer  lan¬ 
guage,  this  test  result  shows  that  die  EVM 
methods  do  not  produce  duration  fore¬ 
casts  better  than  those  from  ES.  A  test  sta¬ 
tistic  value  outside  of  the  critical  region  is 
the  converse;  that  is,  there  is  not  enough 
evidence  to  reject  the  null  hypothesis. 

The  test  statistic  is  determined  from 
the  ranking  of  the  standard  deviations  for 
each  of  the  forecasting  methods  for  each 
project.  The  standard  deviation  is  calculat¬ 
ed  from  the  differences  between  the  fore¬ 
cast  values  computed  at  the  project  status 
points  and  the  actual  final  duration  as 
defined: 


Om  =  [X  (FVm(i)  -  FD)2  /  (n-1)]05 
where 

Cm  =  Standard  deviation  for  forecasting 
method  m 

FVm(i)  =  Forecast  value  for  method  m  at 
status  point  (i) 

FD  =  Actual  final  duration 
n  =  Number  of  status  points 
X  =  Summation  over  a  specific  set  of  sta¬ 
tus  points 

The  smallest  value  for  the  standard 
deviation  indicates  the  best  forecast  pro¬ 
duced.  There  are  five  forecasting  methods 
ranking  from  1  to  5,  with  1  associated  with 
the  lowest  and  5  the  highest  value  of  stan¬ 
dard  deviation.  The  ranking  of  the  meth¬ 
ods  is  performed  for  the  16  sets  of  project 
data.  The  number  of  times  the  rank  of  1 


occurs  (without  ties  between  the  EVM 
methods)  determines  the  test  statistic 
value.  By  using  the  ranking  approach,  the 
unit  for  the  periods  (e.g.,  months,  weeks) 
can  be  different  between  projects;  the 
ranking  of  the  five  methods  is  performed 
separately  for  each  project. 

To  understand  whether  a  particular 
method  is  better  for  early,  middle,  late,  or 
overall  forecasting,  the  projects  are  ana¬ 
lyzed  and  tested  for  specific  regions  of 
performance.  Groupings  are  formed  using 
the  observations  within  various  percent 
complete  ranges  to  make  the  determina¬ 
tions:  early  (10-40  percent),  middle  (40-70 
percent),  late  (70-100  percent),  overall  (10- 
100  percent).  Additionally,  other  ranges 
are  used  to  determine  if  one  of  the  meth¬ 
ods  converges  to  the  actual  final  duration 
more  rapidly  than  the  others,  thus  being 
better  for  a  portion  of  the  forecast  (but 
not  necessarily  superior  overall).  The 
ranges  used  for  this  purpose  are:  25-100 
percent,  50-100  percent,  and  75-100  per¬ 
cent. 

Data  Discussion 

A  total  of  16  projects  are  included  in  the 
study.  Twelve  (1  through  12)  are  from  one 
source  with  four  (13  through  16)  from  a 
second.  The  output  of  the  12  projects  is 
high  technology  products.  The  remaining 
four  projects  are  associated  with  IT  prod¬ 
ucts. 

The  primary  data  requirement  is  that 
the  projects  used  in  the  study  have  not 
undergone  any  re-planning.  The  require¬ 
ment  is  necessary  to  be  able  to  discern  the 
ability  of  the  forecasting  methods  without 
having  outside  influence.  All  16  projects 
performed  from  beginning  to  completion 
without  having  baseline  changes. 

Table  1  (see  page  11)  illustrates  the 
schedule  performance  of  the  projects  in 
the  data  set.  The  12  high  technology  pro¬ 
jects  are  measured  in  monthly  periods 
whereas  the  four  IT  projects  are  measured 
weekly.  Two  projects  were  completed 
early,  three  on  time,  and  the  remaining  11 
later  than  planned. 

Results  Analysis 

To  begin  the  analysis,  it  is  instructive  to 
view  the  graphs  from  a  single  project 
(Project  #13).  The  first  graph,  Figure  2, 
portrays  the  forecasting  performance  of 
all  five  methods  along  with  the  horizontal 
line  for  the  actual  final  duration.  It  is 
observed  that  the  prediction  using  the 
PVav  and  EVav  Work  Rates  behave  in  a 
much  less  erratic  manner  than  do  the  fore¬ 
casts  from  the  current  period  rates,  PVlp 
and  EVlp.  The  forecast  from  ES  is  seen  to 
be  much  better  than  any  of  the  EVM  pre- 


Figure  3:  Time  Forecasting  Standard  Deviation  Comparisons 
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Figure  4:  Comparison  of  Forecasting  Accuracy 


dictions,  especially  after  the  project  com¬ 
pletion  point  of  40  percent. 

The  next  graph,  Figure  3,  portrays 
similar  information.  It  contains  plots  of 
the  standard  deviation  versus  percent 
complete  for  each  of  the  EVM  and  ES 
methods.  The  behavior  seen  in  Figure  2  is 
amplified  by  viewing  the  standard  devia¬ 
tion.  As  described  for  Figure  2,  the  aver¬ 
age  work  rates  are  less  volatile,  while  the 
current  rates  have  large  changes  from  one 
observation  to  the  next.  Again,  the  ES 
forecast  is  observed  to  be  much  more  sta¬ 
ble  than  any  of  the  other  methods.  The 
standard  deviation  of  the  ES  forecast  is 
noticeably  smaller  than  any  of  the  other 
methods  between  50  and  100  percent 
complete. 

Figure  4  is  a  column  graph  illustrating 
a  view  intended  for  analyzing  the  forecast¬ 
ing  behavior  for  early,  middle,  late,  and 
overall  ranges  of  project  execution.  Figure 
5  is  also  a  column  graph;  the  ranges 
applied  (25-100  percent,  50-100  percent, 
75-100  percent)  are  used  to  determine  the 
behavior  of  the  various  methods  regard¬ 
ing  the  rate  of  convergence  to  the  final 
duration. 

For  both  Figures  4  and  5,  it  is  clearly 
seen  that  the  current  period  methods  are 
generally  more  volatile  and  that  the  ES 
method  is  the  better  predictor  in  every 
range.  In  fact,  for  this  project,  the  accura¬ 
cy  of  the  ES  forecasting  method  is  signif¬ 
icantly  better  than  the  EVM  methods. 

It  was  previously  mentioned  that,  with 
the  exception  of  the  forecast  using  the 
PVIp  Work  Rate,  all  of  the  other  methods 
converge  to  calculate  die  actual  final  dura¬ 
tion.  Because  of  this  characteristic,  the 
expectation  is  that  the  standard  deviation 
should  decrease  as  the  completion  per¬ 
centage  increases.  This  behavior  is 
observed  for  ES  and  EVIp,  but  the  others 
are  nearly  invariant  between  the  25-100 
percent,  50-100  percent,  and  75-100  per¬ 
cent  ranges.  Looking  back  at  Figure  3,  the 
convergence  is  seen  for  PVav  and  EVav, 
but  it  is  not  strongly  evident  until  after  the 
project  has  progressed  past  80  percent 
complete. 

Table  2  (see  next  page)  is  an  example 
for  the  10-40  percent  completion  range.  It 
is  the  tabulation  of  the  computed  stan¬ 
dard  deviations  for  each  forecasting 
metiiod  and  their  ranking  for  each  project. 
From  reviewing  the  table,  an  observation 
is  made  that  for  this  completion  range,  the 
ES  rank  is  1  for  11  of  the  projects.  The  ES 
forecasting  method  provides  the  best 
forecasts  of  final  duration  for  a  large 
majority  of  the  projects.  Even  so,  it  does 
not  produce  the  best  forecast  results  for 
every  project.  All  seven  ranges  are  ana¬ 


lyzed  to  understand  more  completely  how 
the  various  methods  perform  under  dif¬ 
ferent  circumstances. 

To  better  understand  the  goodness  of 
the  forecasting  methods  for  the  examined 
completion  band,  Table  3  (see  next  page) 
was  created.  It  is  a  condensation  of  Table 
2.  As  can  be  observed,  the  distribution  of 
the  ranking  numbers  is  made  between  the 
various  forecasting  methods.  In  general, 
die  sum  for  each  of  the  ranks  will  equal  the 
number  of  projects,  16.  However,  when 
there  are  ties,  as  there  is  for  dais  range,  one 
rank  may  total  more  than  16  while  an  adja¬ 
cent  rank  will  be  equally  lower.  For  Table  3, 


it  is  noted  the  sum  of  the  Is  is  seventeen, 
while  the  sum  of  the  2s  is  15. 

At  die  bottom  of  Table  3,  a  weighted 
average  of  the  ranking  distribution  is 
computed  for  each  of  the  forecasting 
metiiods.  These  weighted  averages  are 
then  used  to  rank  the  methods  for  the 
completion  range  examined.  Table  4  (see 
next  page)  is  a  tabulation  of  the  weighted 
averages  of  the  rankings  for  each  of  the 
seven  completion  ranges.  For  each  range, 
the  ES  mediod  has  the  lowest  weighted 
average,  indicating  that,  on  average,  it  is 
the  best  predictor  of  final  duration.  The 
only  challenge  to  ES  is  within  the  40-70 


Figure  5:  Forecasting  Convergence 
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Project  ID 

Project  #1 

Project  #2 

Project  #3 

Project  #4 

Project  #5 

Project  #6 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

PVav 

14.95 

5 

13.01 

4 

11.93 

2 

25.59 

2 

4.38 

2 

29.76 

2 

EVav 

2.65 

1 

9.35 

2 

8.28 

1 

48.68 

4 

5.82 

3 

42.64 

4 

Methods 

PVIp 

5.47 

2 

13.62 

5 

77.74 

5 

42.77 

3 

8.67 

4 

42.11 

3 

EVIp 

6.00 

3 

12.14 

3 

22.38 

3 

103.15 

5 

9.89 

5 

263.03 

5 

ES 

8.28 

4 

4.78 

1 

46.76 

4 

14.03 

1 

1.88 

1 

3.57 

1 

Proje 

ct  ID 

Proje 

ct  #7 

Proje 

ct  #8 

Proje 

O 

CO 

Proje 

Dt  #10 

Proje 

ct  #1 1 

Projec 

t  #12 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

PVav 

9.79 

3 

16.16 

3 

6.75 

2 

9.06 

1 

7.66 

4 

15.06 

3 

EVav 

6.00 

2 

33.17 

5 

15.63 

3 

10.55 

2 

6.63 

3 

30.49 

5 

Methods 

PVIp 

17.95 

5 

20.69 

4 

20.80 

4 

39.11 

4 

7.70 

5 

9.06 

1 

EVIp 

15.07 

4 

5.69 

2 

525.62 

5 

102.21 

5 

6.58 

2 

26.86 

4 

ES 

4.31 

1 

5.09 

1 

3.74 

1 

15.22 

3 

4.54 

1 

12.49 

2 

Proje 

ct  ID 

Proje 

ct  #13 

Proje 

ct  #14 

Proje 

ct  #15 

Proje 

Dt  #16 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

Std  Dev 

Rank 

PVav 

10.57 

2 

2.36 

1 

15.93 

3 

20.18 

5 

EVav 

22.78 

3 

5.90 

5 

18.12 

5 

17.10 

4 

Methods 

PVIp 

28.25 

4 

2.36 

1 

11.24 

2 

12.37 

2 

EVIp 

33.59 

5 

2.49 

4 

16.87 

4 

16.49 

3 

ES 

8.62 

1 

4.46 

3 

4.45 

1 

5.20 

1 

Table  2:  Standard  Deviation  and  Ranking  for  10-40  Percent  Completion  Range 


percent  middle  range,  where  the  weighted 
average  of  2.063  for  ES  is  somewhat 
lower  than  the  2.500  from  PVav. 

Finally,  more  conclusive  evidence  of 
the  goodness  of  the  ES  forecasting  capa¬ 
bility  is  provided  from  the  statistical 
hypothesis  testing.  Table  5  provides  the 
compiled  results  from  the  testing  analysis. 


In  the  table,  the  count  of  the  rank  of  1  is 
provided  for  the  aggregate  of  the  EVM 
methods  and  for  ES.  With  the  exception 
of  one  test  range,  ES  shows  to  be  superi¬ 
or  to  the  other  methods  combined.  In 
one  instance,  the  40-70  percent  range,  the 
number  of  Is  counted  for  EVM  exceeds 
the  number  for  ES.  However,  the  value  of 


the  test  statistic  is  in  the  critical  region; 
this  is  enough  evidence  to  reject  the  null 
hypothesis  that  the  aggregate  of  the 
EVM  methods  is  better  than  the  ES 
method.  Thus,  from  the  results  of  the 
Sign  Test,  ES  is  indicated  to  be  the  better 
forecasting  method  regardless  of  project 
completion  stage  (early,  middle,  late,  and 
overall) . 

Summary  and  Conclusions 

Five  methods  of  project  duration  fore¬ 
casting  were  examined  in  this  study,  four 
from  EVM  and  the  ES  technique. 
Performance  data  from  16  projects  was 
used  to  assess  the  capabilities  of  the  vari¬ 
ous  forecasting  methods.  The  analysis 
strategy  segregated  the  project  data  into 
seven  ranges  of  percent  complete  in  order 
to  isolate  possible  forecasting  characteris¬ 
tics  or  tendencies  among  the  methods. 

Each  of  the  methods  were  used  to 
create  forecasts  from  the  project  data. 
The  standard  deviation  of  the  forecasts 
from  the  actual  final  duration  was  com¬ 
puted  for  each  project,  and  each  percent 
complete  range  was  studied.  The  fore¬ 
casting  methods  were  then  ranked  from 
best  to  worst  using  the  standard  devia¬ 
tions. 

The  tabulation  of  best  forecasts  (one 
of  the  four  EVM  methods  or  ES)  for 
each  range  was  used  to  calculate  the  test 
statistic  for  the  Sign  Test.  The  null 
hypothesis — that  EVM  provides  the  bet¬ 
ter  forecast — was  rejected  for  every  per¬ 
cent  complete  range  examined. 

Conclusively,  from  among  the  meth¬ 
ods  and  data  set  studied,  ES  is  shown  to 
be  the  best  method  of  forecasting  project 
duration. ♦ 


Table  3:  Rank  Count  for  Data  Group,  10-40  Percent 


Methods 

PVav 

EVav 

PVIp 

EVIp 

ES 

Count 

Nrls 

2 

2 

2 

0 

11 

Nr2s 

6 

3 

3 

2 

1 

Nr3s 

4 

4 

2 

4 

2 

Nr4s 

2 

3 

5 

4 

2 

Nr5s 

2 

4 

4 

6 

0 

Weighted  Average 

2.750 

3.250 

3.375 

3.875 

1.688 

Composite  Rank 

2 

3 

4 

5 

1 

Table  4:  Weighted  Average  of  Rain  king  Results 


Percent  Complete  Test  Bands 

10%-40% 

40%-70% 

70%-100% 

10%-100% 

25%-100% 

50%-100% 

75%-100% 

ES 

1.688 

2.063 

1.438 

1.625 

1.563 

1.563 

1.438 

PVav 

2.750 

2.500 

3.688 

2.625 

2.813 

3.063 

3.875 

EVav 

3.250 

2.813 

2.938 

3.00 

3.063 

2.938 

2.875 

PVIp 

3.375 

3.438 

3.875 

3.813 

3.875 

3.688 

3.875 

EVIp 

3.875 

4.188 

3.063 

3.938 

3.688 

3.750 

2.938 

Table  5:  Hypothesis  Test  Results  —  El  DA  vs.  ES  Time  Forecast 


Significance 
a  =  0.05 

Percent  Complete  Test  Bands 

10%-40% 

40%-70% 

70%-1 00% 

10%-100% 

25%-100% 

50%-1 00% 

75%-100% 

Test  Statistic 
Sign  Test 

0.0000 

0.0267 

0.0000 

0.0000 

0.0000 

0.0002 

0.0000 

Ha 

Ha 

Ha 

Ha 

Ha 

Ha 

Ha 

Counts  ES 

11 

7 

12 

11 

11 

10 

12 

#1s  EVM 

5 

9 

4 

5 

5 

6 

4 

Hypothesis  Test:  Sign  Test  at  0.05  level  of  significance 

Ho:  The  aggregate  of  EVM  forecasts  is  better/the  null  hypothesis 

Ha:  ES  forecast  is  better/the  alternate  hypothesis 
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Web  Sites 


Earned  Value  Management  (EVM) 

www.  earnedvaluemanagement.  com 

Learn  more  about  the  project  management  system  that  combines 
schedule  performance  and  cost  performance  to  answer  the  ques¬ 
tion,  “What  did  we  get  for  the  money  we  spent?”  The  Web  site 
describes  the  basic  concepts  of  EVM:  project  steps  earning  value 
as  work  is  completed,  comparing  Earned  Value  to  the  actual  and 
planned  costs  to  determine  project  and  future  performance,  and 
measuring  the  physical  project  progress  in  dollars  so  that  both 
schedule  and  cost  performance  can  be  analyzed  in  the  same 
terms.  The  Web  site  also  details  EVMs  benefits,  its  building 
blocks,  its  performance  indices  and  variance,  its  forecasting  capa¬ 
bilities,  and  ways  to  get  your  organization  or  business  started  in 
utilizing  EVM. 

Earned  Schedule  (ES) 

www.earnedschedule.com 

ES  is  a  breakthrough  analytical  technique  that  resolves  the  EVM 
dilemma  of  schedule  indicators  providing  false  information  for 
late-performing  projects.  It  is  derived  from  and  is  an  extension 
of  EVM,  needing  no  additional  data  for  acquiring  the  ES  mea¬ 
sures  (just  the  data  from  EVM).  Along  with  learning  the  process 
of  using  ES,  this  site  defines  ES  terminology  offers  links  to  the 
latest  ES  news,  publications,  and  presentations,  and  provides  a 
free  downloadable  ES  calculator. 


Get  your  ducks  in  a  row 

www.washingtontechnology.com/print/ 23-02/32228- 1  .html 
To  contractors  and  agencies,  ISO  certifications  and  CMMI  rat¬ 
ings  denote  specific  accomplishments  in  implementing  method¬ 
ical,  disciplined  processes.  In  his  article  for  Washington 
Technology,  Michael  Hardy  argues  that  while  earning  certifica¬ 
tions  is  costly  and  time-consuming,  companies  cannot  avoid 
making  the  investment  if  they  expect  to  remain  competitive.  He 
also  highlights  several  companies,  showing  both  their  processes 
in  earning  ISO  certifications  and  CMMI  Level  ratings,  and  the 
competitive  benefits  those  certifications  have  yielded. 

Guide  to  the  Software  Engineering  Body 
of  Knowledge 

www.swebok.org/ironman/pdf/SWEBOK  Guide  2004.pdf 
For  the  first  time,  the  IEEE  Computer  Society  has  established  a 
baseline  for  the  body  of  knowledge  for  the  field  of  software 
engineering.  The  Guide  does  not  claim  to  define  the  body  of 
knowledge  but  rather  it  serves  as  a  compendium  and  guide  to 
the  body  of  knowledge  that  has  been  developing  and  evolving 
over  the  past  four  decades.  The  Guide  is  subdivided  into  10 
software  engineering  Knowledge  Areas,  including  software 
requirements,  design,  construction,  testing,  maintenance,  con¬ 
figuration  management,  engineering  management,  engineering 
process,  engineering  tools/methods,  and  quality. 
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The  Two  Most  Useful  Earned  Value  Metrics: 

The  CPI  and  theTCPI 

Quentin  W  Fleming  and  Joel  M.  Koppelman 

Primavera  Systems,  Inc. 

The  Project  Management  Institute  (PMI)  has  just  released  the  4  th  edition  of  their  world  standard  on  project  management:  A 
Guide  to  the  Project  Management  Body  of  Knowledge.  Many  new  features  have  been  added  to  this  massive  docu¬ 
ment,  among  them  new  coverage  of  an  Earned  Value  metric  called  the  To-Complete  Performance  Index.  What  is  the  To- 
Complete  Performance  Index  and  why  is  it  important ?  This  article  describes  its  purpose  and  utility,  and  how  it  can  work  with 
the  Cost  Performance  Index. 


Whenever  a  project  commits  to  the 
employment  of  Earned  Value  to 
help  manage  their  effort,  users  are  sud¬ 
denly  inundated  with  a  windfall  of  per¬ 
formance  metrics  which  are  available  in 
no  other  project  management  technique. 
New  acronyms  suddenly  emerge:  PV, 
EV,  AC,  SV,  SPI,  CV,  CPI,  BAC,  EAC, 
TCPI1,  and  on  and  on.  While  all  of  these 
performance  indicators  can  have  value 
to  any  project,  the  two  Earned  Value 
Management  (EVM)  metrics  particularly 
critical  to  projects  are  the  CPI  and  the 
TCPI. 

The  CPI  tells  the  user  what  has  been 
accomplished  for  what  has  already  been 
spent:  Did  the  project  stay  within  the 
budget,  or  was  there  an  overrun?  By 
contrast,  the  TCPI  focuses  on  future 
work  questions  such  as:  What  perfor¬ 
mance  levels  must  be  achieved  on  the 
remaining  work  in  order  to  meet  our 
financial  commitment  to  management? 
While  most  practitioners  of  EV  under¬ 
stand  the  utility  of  the  CPI,  most  have 
rarely  used  the  TCPI.  It’s  a  pity  because 
the  TCPI,  when  used  in  conjunction 
with  the  CPI,  provides  a  powerful  set  of 
tools  in  the  management  of  a  single 
project,  a  program,  or  a  full  portfolio  of 
projects. 

EVMrThe  10  Requirements 

As  a  general  rule,  whenever  a  project 
manager  makes  the  decision  to  employ 
EV  in  the  management  of  a  project,  that 
choice  ideally  should  be  supported  by 
management,  the  stakeholders  at  all  lev¬ 
els.  Stakeholders  must  want  to  know  the 
full  truth.  The  reason?  EVM  perfor¬ 
mance  data  can  be  available  to  everyone 
working  the  project:  the  functions, 
senior  management,  the  paying  cus¬ 
tomers,  and  essentially  all  parties  who 
have  a  vested  interest  in  the  success  of 
the  project.  As  long  as  everyone  has  a 
rudimentary  understanding  of  what  the 
EVM  data  means,  everyone  connected 
to  the  project  knows  what  everyone  else 
is  doing.  Thus,  it  is  imperative  that  there 


be  a  management  buy-in  whenever  a 
project  manager  elects  to  employ  EVM 
on  a  project. 

The  commitment  to  employing  EVM 
requires  both  compliance  with  certain 
basic  requirements  and  discipline  on  the 
part  of  everyone  supporting  the  project. 
Based  on  our  experience,  we  have  listed 
the  following  10  key  requirements  which 
must  be  met  in  order  to  successfully 
implement  EVM.  Some  find  these 
requirements  overwhelming.  See  for 
yourself.  These  requirements  are: 

“ ...  the  TCPI,  when 
used  in  conjunction 
with  the  CPI,  provides 
a  powerful  set  of 
tools  in  the  management 
of  a  single  project, 
a  program,  or  a 
full  portfolio 
of  projects .” 

1.  EVM  requires  that  the  project  be 
fully  understood,  defined,  and 
scoped  to  include  100  percent  of  the 
project  effort.  You  need  to  know 
what  constitutes  100  percent  of  the 
work  in  order  to  measure  progress 
along  the  way. 

2.  EVM  requires  that  the  defined  scope 
be  decomposed.  In  other  words,  the 
scope  is  broken  down  into  major 
management  tasks  that  are  selected 
as  points  of  management  control2, 
then  planned  and  scheduled  down  to 
the  detailed  work  package  level. 

3.  EVM  requires  that  an  integrated  and 
measurable  project  baseline  be 


authorized,  relating  the  scope  of 
work  directly  to  an  achievable  bud¬ 
get,  then  locked  into  a  specific  time- 
frame  for  performance  measure¬ 
ment.  It’s  called  bottom-up  planning. 

4.  EVM  requires  that  only  authorized 
and  budgeted  work  be  accomplished. 
The  effort  being  worked  must  be 
tightly  controlled.  Scope  creep  can¬ 
not  be  allowed.  All  changes  must  be 
managed,  and  not  worked  until 
specifically  authorized  by  the  project 
manager. 

5.  EVM  requires  that  physical  perfor¬ 
mance  be  measured  (the  EV)  using 
previously  defined  schedule  metrics. 

6.  EVM  requires  that  the  values  earned 
be  related  to  the  PVs  to  reflect  per¬ 
formance  against  the  project  base¬ 
line.  EV  less  the  PV  represents  a 
variance  from  the  baseline  plan. 

7.  EVM  requires  that  the  ACs  being 
reported  be  consistent  with  the  EV 
being  measured  to  allow  for  an  accu¬ 
rate  portrayal  of  cost  performance. 
The  relationship  of  values  earned  to 
ACs  reflects  the  true  cost  perfor¬ 
mance.  EV  less  ACs  provides  cost 
performance. 

8.  EVM  requires  that  forecasts  be  made 
periodically  (weekly,  monthly)  as  to 
how  much  time  and  money  it  will 
take  to  complete  100  percent  of  the 
project. 

9.  EVM  requires  that  a  full  disclosure 
of  actual  results  be  made  to  all  per¬ 
sons  who  have  a  vested  interest  in 
the  project.  All  stakeholders  will 
receive  the  same  actual  performance 
results.  Only  one  set  of  books  is 
allowed. 

10.  EVM  requires  that  project  manage¬ 
ment,  in  conjunction  with  manage¬ 
ment  at  all  levels  and  customer  stake¬ 
holders,  decide  on  the  appropriate 
actions  to  be  taken  to  stay  within  the 
authorized  project  expectations. 
These  10  requirements  are  needed  to 

successfully  implement  EV  on  any  pro¬ 
ject.  In  our  opinion,  these  requirements 
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1.00 


_ Good  =  greater  than  1.0 

- ►  Perfect  CPI  =  1.0 


. *  Poor  =  less  than  1.0 


Perfect  EVMCPI  =  1.0 

(for  every  $1.00  in  cost  actuals  you  get  $1.00  in  EV) 


Figure  1 :  Monitoring  Earned  l  ralue  Cost  Performance 


constitute  nothing  more  than  following 
fundamental  project  management  best 
practices. 

We  will  now  discuss  what  we  believe 
to  be  the  most  important  EV  indicators: 
the  CPI  reflecting  completed  perfor¬ 
mance,  and  the  TCPI  with  a  focus  on 
the  required  future  performance. 

What  Is  a  CPI,  and  How  Is  it 
Used? 

The  EVM  CPI  is  a  reflection  of  project 
cost  efficiency.  The  CPI  relates  the 
physical  work  accomplished,  expressed 
in  its  budgeted  value,  against  the  ACs 
incurred  to  accomplish  the  performed 
work.  Budgets  can  be  set  with  various 
monetary  values,  hours,  deliverables,  or 
anything  else  that  can  be  measured.  The 
issue:  Is  the  project  staying  on  target, 
underrunning,  or  perhaps  overrunning 
costs?  This  concept  is  portrayed  in 
Figure  1. 

Perfect  cost  performance  would  be 
defined  as  achieving  a  CPI  of  1.0:  For 
every  dollar  spent,  we  would  get  an  EV 
equal  to  one  dollar.  Sometimes  we  do 
better,  sometimes  worse.  This  is  a  par¬ 
ticularly  critical  metric  to  track  because 
performance  at  less  than  1.0  is  a  reflec¬ 
tion  of  excessive  costs  spent  against 
budget.  Initial  overruns  are  typically 
non-recoverable.  Think  about  it:  In 
order  to  recover  an  overrun,  future 
work  must  be  underrun.  Rarely  does  this 
happen.  The  same  conditions  which 
may  have  caused  the  overrun  in  the  first 
place  are  likely  to  occur  again  and  again. 

Sometimes  the  CPI  will  reflect  val¬ 
ues  over  1.0,  suggesting  that  an  under- 
run  of  costs  is  occurring.  Care  must  be 
taken  when  actuals  reflect  an  underrun 
of  costs  to  budget.  Oftentimes,  this 
condition  is  simply  the  result  of  costs 
which  are  lagging  (slow  to  be  recorded 
in  the  organizational  cost  ledger).  For 
example,  let’s  say  you  measure  the  EV 
and  give  full  credit,  but  the  related  costs 
are  not  reflected  in  the  cost  ledger.  The 
reason?  Most  of  the  project  work  may 
be  performed  by  outside  purchased 
labor  (people  who  are  not  part  of  the 
internal  labor  system).  Thus,  there  can 
be  a  time  mismatch  between  the  EV 
measured  and  the  actual  payment  of  the 
purchased  labor  invoices.  The  payment 
of  invoices  generally  takes  more  time 
than  the  recording  of  labor. 

Underruns  of  costs  are  rare.  And,  if 
artificially  caused  by  lagging  ACs,  the 
positive  results  can  hide  or  offset  prob¬ 
lems  that  need  management  attention.  It 
takes  organizational  discipline  to  make 


sure  that  EV  credits  match  the  ACs. 

Why  is  the  CPI  so  important? 
Because  past  performance  can  be  used 
to  accurately  determine  requirements 
for  final  performance,  in  order  to  meet 
financial  goals.  The  cumulative  (from 
the  beginning)  CPI  has  been  shown  to 
stabilize  from  about  the  20  percent 
completion  point  of  project  perfor¬ 
mance.  Empirical  scientific  studies  by 
the  DoD  on  155  actual  contracts  has 
shown  that  at  the  20  percent  point  of 
project  completion,  the  final  projected 
results  will  only  change  by  plus  or 
minus  10  percent  [1J.  What  a  finding! 
What  useful  data. 

In  practical  terms,  one  can  immedi¬ 
ately  take  the  total  authorized  budget 
(BAC),  divide  it  by  the  cumulative  CPI, 
and  predict  the  total  costs  of  a  project 


with  an  accuracy  of  plus  or  minus  10 
percent.  If  management  doesn’t  like  the 
final  cost  projection,  then  corrective 
action  can  be  taken  to  change  the  fore¬ 
casted  results.  Few  project  management 
techniques  give  a  comparable  early- 
warning  signal.  This  formula,  the 
BAC/ Cumulative  CPI  =  EAC,  can  be 
used  on  the  total  project,  or  any  sub- 
project,  or  with  integrated  project  teams 
to  predict  final  results  on  their  work. 

The  CPI  metric  can  be  used  to  track 
periodic  results  (monthly,  weekly,  daily) 
or  the  cumulative  position  to  see  the 
long-term  performance  trends. 

What  Is  aTCPI,and  How  Is 
it  Used? 

Whereas  the  CPI  is  an  indicator  of  past 
cost  performance,  the  TCPI  has  its 


Figure  2:  The  Relationship  of  Cumulative  CPI  vs.  TCPI 
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Data  and  Data  Management 


TCPI  using  Management’s  “Budget  at  Completion”  (BAC): 


Work  Remaining  (BAC  -  EV) 
Funds  Remaining  (BAC  -  AC) 

/ 


-  TCPI  (BAC) 


TCPI  using  the  Project  Manager’s  “Estimate  at  Completion”  (EAC) 


Work  Remaining  (BAC  -  EV) 
Funds  Remaining  (EAC  -  AC) 


=  TCPI  (EAC) 


Figure  3:  Two  TCPI  Formulas 


focus  on  future  performance.  At  issue: 
What  will  it  take  to  meet  the  goals  set  by 
management?  The  TCPI  works  in  con¬ 
junction  with  the  CPI,  and  is  conceptual¬ 
ly  illustrated  in  Figure  2  (see  previous 
page). 

The  CPI  can  be  thought  of  as  sunk- 
costs;  whatever  the  results,  they  cannot  be 
altered.  In  the  illustration  shown,  the 
cumulative  CPI  is  at  .75;  for  every  dollar 
spent,  only  75  cents  of  project  work  was 
earned.  If  the  project  is  exactly  50  per¬ 
cent  complete,  one  would  need  to  accom¬ 
plish  $1.25  for  every  future  dollar  in  order 
to  stay  within  management’s  budget.  Will 
this  happen?  At  best,  it  is  highly  unlikely. 
Opportunities  for  improvements  are 
illustrated  by  the  use  of  the  TCPI. 

The  formula  for  the  TCPI  is:  The 
[Work  Remaining]  (defined  as  total 
Budget  less  the  EV)  divided  by  the 
[Funds  Remaining],  Note  that  in  Figure  3 
there  are  two  scenarios  for  Funds 
Remaining3.  Funds  remaining  will  focus 
initially  on  the  authorized  budget. 
Management  will  track  performance 
against  what  it  has  authorized  in  the  form 
of  an  official  budget.  However,  once  it 
becomes  obvious  that  the  budget  is  no 
longer  achievable,  management  must 
determine  how  much  money  it  will  cost 
to  complete  this  job  (called  the  EAC). 
The  project  then  stops  work  and  makes  a 
new  forecast  of  what  is  needed  to  finish 
the  job. 

Preparing  a  new  EAC  forecast  can  get 
emotional.  Unrealistic  optimism  some¬ 
times  takes  over,  at  the  expense  of  real¬ 
ism.  It  is  not  uncommon  for  projects, 
when  making  a  new  EAC  forecast,  to 
assume  that  everything  will  suddenly  go 
right,  and  that  all  project  risks  are  behind 


them.  Thus,  an  initial  EAC  may  be  unre¬ 
alistic  or  unachievable.  Piecemeal  EACs 
are  often  the  norm,  where  the  EAC  pro¬ 
jection  goes  up  each  month  as  actual  per¬ 
formance  is  known. 

Using  Figure  2  as  an  example,  would 
an  EAC  requiring  a  future  TCPI  of  1.25 
or  1.10  be  achievable?  Probably  not. 
More  likely,  a  TCPI  of  1.0  or  .90  would 
be  reasonable.  But  it  is  painful  to  admit 
the  full  value  of  an  EAC,  having  just 
acknowledged  that  the  BAC  is  no  longer 
valid. 

Conclusion 

Employing  the  EVM  technique  can  pre¬ 


sent  a  project  with  data  not  available 
with  any  other  management  tool.  And 
while  each  metric  can  be  useful,  we 
believe  that  the  two  metrics  described 
are  particularly  useful  in  the  manage¬ 
ment  of  any  project,  or  program,  or  a 
portfolio  of  projects. 
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Notes 

1.  The  terms  are:  Planned  Value, 
Earned  Value,  Actual  Cost,  Schedule 
Variance,  Schedule  Performance 
Index,  Cost  Variance,  Cost  Perfor¬ 
mance  Index,  Budget  at  Completion, 
Estimate  at  Completion,  and  To- 
Complete  Performance  Index.  All 
terms  used  in  this  article  are  consis¬ 
tent  with  the  “Guide  to  the  Project 
Management  Body  of  Knowledge,” 
4th  edition,  published  in  December 
2008  by  the  PMI. 

2.  The  points  of  management  control 
are  sometimes  called  project  teams, 
subprojects,  or  control  account 
plans,  depending  on  the  organiza¬ 
tion. 

3.  Figures  used  in  this  article  are 
inspired  by  “Earned  Value  Project 
Management,”  3rd  edition,  Quentin 
W  Fleming  and  [oel  M.  Koppelman, 
PMI,  2005. 
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Certifications  Help  Organizations  and  Clients 

George  Jackelen 
Software  Consultants,  Inc. 

As  part  of  the  United  States  government’s  request for  proposal  (RFP)  process  to  acquire  products  and  services,  more  empha¬ 
sis  is  currently  being  placed  on  the  past  pe  formance /  experience  proposal  section.  This  article  addresses  another  area  the  gov¬ 
ernment  (and  other  potential  clients)  should  examine,  especially  for  potential  bidders  with  limited  experience:  organisational 
certification  by  certified,  independent  assessors  based  on  internationally  and  nationally  recognised  standards  and  methodologies' . 


In  a  Washington  Technology  article  [1], 
Michael  Hardy  describes  how  certifica¬ 
tions  help  contractors  to  compete  and 
thrive.  This  article  expands  this  theme  by 
describing  how  contractor  certifications 
can  help  organizations  improve,  help 
clients  choose  contractors,  and  how  certi¬ 
fications  can  be  used  to  provide  better  ser¬ 
vices  and  products. 

My  definition  of  certification  is: 

Being  accredited  by  an  external 
independent  group  certified  by  a 
standard’s  (to  include  methodolo¬ 
gies)  owner  to  evaluate  organiza¬ 
tions  and  to  provide  objective  evi¬ 
dence  to  a  nationally/international¬ 
ly  recognized  group  (for  example, 
the  American  National  Standards 
Institute  —  American  Society  for 
Quality  National  Accreditation 
Board)  to  issue  formal  certificates 
of  approval. 

Internal  accreditation  is  not  addressed 
because,  in  my  opinion,  they  rarely  pro¬ 
vide  independence  to  identify  and  docu¬ 
ment  the  existence  of  objective  evidence 
and  artifacts;  indeed,  I  have  witnessed  data 
manipulation  to  provide  an  organization 
with  the  desired  results.  I  do  accept  the 
importance  of  internal  audits  for  the  pur¬ 
pose  of  gap  analysis  to  determine  what  is 
missing  to  fulfill  compliance  requirements. 

Capacity  and  Experience 

Before  going  into  a  discussion  about  cer¬ 
tifications,  I  want  to  discuss  the  terms 
capacity  and  experience ,  which  some  people 
consider  to  be  the  same. 

An  organization’s  capacity — defined  as 
“the  potential  or  suitability  for  holding, 
storing,  or  accommodating”  [2] — is  not  a 
reliable  evaluation  tool.  Capacity  does  not 
always  provide  objective  evidence  that  an 
organization  can  actually  provide  this 
capacity  or  really  knows  how  to  satisfy  the 
requirements.  Thus,  an  organization  can 
have  a  capacity  to  do  something  without 
ever  having  experience  doing  that  tiling. 
Also,  a  stated  capacity  may  only  be  due  to 


one  employee’s  capacity  (who  may  leave 
the  organization  or  may  not  work  on  the 
contract)  or  education  (e.g.,  a  person  took 
a  course  on  the  topic,  but  has  no  applica¬ 
tion  experience);  rather  than  having  sever¬ 
al  people  with  the  capacity  and/ or  having 
organizational  documented  and  imple¬ 
mented  procedures  on  how  to  provide  a 
stated  capacity.  Thus,  there  may  be  no  evi¬ 
dence  to  show  a  capacity  was  ever  provid¬ 
ed  successfully  by  an  organization. 

“I  have  seen  the 
distinction  between 
capacity  and  experience 
applied  to  RFPs  when  a 
client  wants  organization/ 
team  experiences 
(reality)  rather  than 
capabilities  (theory).” 

An  organization’s  experience — defined 
as  “the  act  of  living  through  an  event  or 
events”  [3] — is  more  valuable  to  a  client. 
Thus,  a  capacity  is  not  the  same  as  experi¬ 
ence,  nor  should  capacity  have  as  much 
weight  in  an  evaluation  as  experience.  For 
example,  advertisements  say  my  car  has 
the  capacity  to  provide  30  miles  per  gallon 
(mpg) ;  but,  in  the  real  world,  I  have  no 
experience  where  my  car  had  an  actual 
performance  measurement  of  30  mpg  or 
greater. 

Thus,  experience  is  better  than  capaci¬ 
ty  to  determine  if  an  organization  can 
support  a  client.  This  is  especially  true  if 
the  client  contacts  tire  referenced  clients, 
identified  in  the  experience  part  of  a  pro¬ 
posal,  for  their  view  of  how  well  the  orga¬ 
nization  performed  and/or  implemented 
their  processes  and  developed  the  needed 
product  or  service.  This  step  is  similar  to 


verifying  a  future  employee’s  references, 
experience,  and  education. 

I  have  seen  the  distinction  between 
capacity  and  experience  applied  to  RFPs 
when  a  client  wants  organization/team 
experiences  (reality)  rather  than  capabili¬ 
ties  (theory)2.  I  recognize  that  experience 
may  not  be  a  true  reflection  of  an  organi¬ 
zation’s  present  and  future  environment. 
Also,  some  organizations  are  too  new  or 
too  small  to  have  the  needed  experience. 
In  these  situations,  an  organization  may 
not  bid  or  a  client  may  not  have  high  con¬ 
fidence  that  an  organization  can  deliver 
what  a  client  expects. 

A  possible  solution  to  this  dilemma  is 
for  clients  to  examine  an  organization’s 
certifications.  I  am  ignoring  employee  cer¬ 
tification  since  people  can  leave  an  organi¬ 
zation  and  employee  certifications  do  not 
show  that  an  organization  has  implement¬ 
ed  the  principles  of  these  certifications. 
However,  I  have  seen  RFPs  requiring  the 
proposed  people  who  will  work  a  contract 
to  have  specific  certifications  (e.g,  related 
to  information  or  computer  security). 

Independent  Certifications 

I  recommend  clients  require  a  copy  of 
each  organizational  certificate  related  to 
the  RFP  Independent  certifications  (e.g, 
the  International  Organization  for  Stan¬ 
dardization  [ISO])  can  help  organizations 
and  clients  reduce  the  risk  of  having  a  lack 
of  experience  by  showing  clients  the  orga¬ 
nization  has  a  certified  set  of  processes  in 
place.  Besides  having  processes  in  place, 
certifications  are  based  on  independently 
observed  objective  evidence  showing  that 
the  processes  are  implemented  as  stated. 

Why  should  clients  believe  that  certifi¬ 
cation  is  a  bridge  between  an  organiza¬ 
tion’s  capacity  and  its  lack  of  client- 
required  experience?  Since  receiving  an 
organizational  certificate  is  not  cheap  and 
cannot  normally  be  obtained  in  a  few 
months,  clients  should  recognize  an  orga¬ 
nization  for  its  willingness  to  expand 
resources  so  they  can  prove  their  process¬ 
es  are  established  and  maintained,  and  are 
actually  implemented.  At  the  same  time 
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(prior  to  being  certified),  auditors /certi¬ 
fiers  spend  a  lot  of  time  looking  for  objec¬ 
tive  evidence  drat  organizations  comply 
with  the  given  standards  and  certification 
requirements.  For  a  client,  this  means 
organizations  must  not  only  have  process¬ 
es  in  place,  but  must  also  prove  these 
processes  are  implemented  as  stated. 

Another  organizational  factor — an 
important  cost-benefit  determination  for 
an  organization — is  deciding  what  part  of 
die  organization  is  to  be  certified  (i.e.,  the 
whole  organization  or  an  organizational 
subset).  Can  an  organization  afford  to  wait 
for  a  payback  that  may  not  appear  for 
months  after  certification?  For  a  client, 
certification  may  be  with  an  organization¬ 
al  subset  that  is  not  proposed  to  partici¬ 
pate  on  a  contract  or  is  only  providing 
minor  contract  support. 

Therefore,  organizations  need  to  make 
a  decision  about  their  need  for  certifica¬ 
tions,  what  certifications  they  want  to 
achieve,  what  part  of  the  organization  to 
certify,  and  their  willingness  to  pay  the 
cost.  Organizations  must  also  be  aware 
that  the  cost  to  be  certified  does  not  end 
with  certification.  For  ISO  and  the 
Software  Engineering  Institute  (SEI),  for 
instance,  achieving  certification  is  not  a  be 
all  and  end  all.  ISO  and  SEI  require  period¬ 
ic  recertification  to  assure  the  certification 
standards  and  organizational  processes  are 
maintained  and  the  processes  are  imple¬ 
mented.  To  an  organization,  achieving  cer¬ 
tification  and  recertification  may  be  a  key 
to  future  contracts,  especially  for  RFPs 
requiring  particular  certificates. 

For  clients,  this  means  certification  is 
not  a  lifelong  license  to  brag  based  on  a  one¬ 
time  evaluation  of  an  organization.  As  a 
result,  clients  need  to  know  when  an  orga¬ 
nization  was  last  certified  and  the  certifi¬ 
cate’s  duration. 

What  Certifications  Will  Meet 
an  Organization’s  Needs? 

Given  what  I’ve  just  explained,  what  cer¬ 
tificates  should  an  organization  apply  for 
and  what  certificates  should  a  client  look 
for?  The  answer  depends  on  an  organiza¬ 
tion’s  goals  and  objectives — and  what  a 
client  is  looking  for  to  ensure  the  right 
organization  is  picked  to  execute  a  con¬ 
tract3. 

Table  1  provides  examples  of  four 
major  international  standards  that  provide 
recognized  certifications,  and  examples  of 
how  these  standards  relate.  The  first  qual¬ 
ification  (ISO  27000)  should  be  strongly 
considered  by  clients  and  organizations 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark 
Office  by  Carnegie  Mellon  University. 
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since  sensitive  data  security  (e.g.,  payroll  or 
personnel  records)  is  critical  to  most 
clients  and  organizations.  In  addition,  this 
certification  is  important  for  its  guidance 
on  providing  physical  and  procedural  pro¬ 
tection  of  data,  physical  equipment,  peo¬ 
ple,  and  the  operational  environment. 

The  second  qualification  (ISO  9001)  is 
arguably  the  standard  that  set  the  standard  for 
die  other  qualifications.  The  third  qualifi¬ 
cation  (ISO  20000)  is  not  known  by  many 
organizations,  but  it  expands  ISO  9001  by 
addressing  IT’s  involvement  with  business 
needs  and  strategy.  Several  ISO  20000 
requirements  relate  to  ISO  27000  and  ISO 
9001 .  As  a  result,  achieving  ISO  20000  cer- 


“ ...  organizations  need 
to  make  a  decision 
about  their  need  for 
certifications ,  what 
certifications  they  want 
to  achieve,  what  part 
of  the  organization  to 
certify,  and  their 
willingness  to  pay 
the  cost.” 


tification  helps  an  organization  to  also 
achieve  ISO  9001  and  ISO  27000  certifica¬ 
tion.  ISO  20000  certification  can  also  help 
organizations  with  CMMI®  appraisals. 

Table  1  also  shows  some  of  tire  simi¬ 
larities  between  the  three  international 
standards  and  an  internationally  accepted 
methodology/model  (CMMI)  to  improve 
quality. 

Certifications 

Due  to  similarities  with  certification,  this 
article  addresses  only  the  following  stan¬ 
dards  because  they  are  internationally 
well-known  in  assisting  organizations  to 
improve  their  quality,  efficiency,  and  effec¬ 
tiveness  (other  standards  could  be  added): 

•  ISO  27000:2005,  IT  -  Security 
Techniques  —  Information  Security 
Management.  Provides  a  model  for 
establishing,  implementing,  operating, 
monitoring,  reviewing,  maintaining, 
and  improving  an  Information  Se¬ 
curity  Management  System  (ISMS). 

•  ISO  9001:2000,  Quality  Manage¬ 
ment  Systems  —  Requirements. 


Intended  for  use  in  any  organization 
which  designs,  develops,  manufac¬ 
tures,  installs,  and/or  services  any 
product  or  provides  any  form  of  ser¬ 
vice.  It  identifies  the  requirements  an 
organization  needs  to  fulfill  to  achieve 
customer  satisfaction  through  consis¬ 
tent  products  and  services  meeting 
client  expectations.  It  includes  a  need 
for  continual  (i.e.,  planned)  improve¬ 
ment  of  a  Quality  Management 
System. 

•  ISO  20000:2005,  IT  -  Service 
Management.  Promotes  the  adop¬ 
tion  of  an  integrated  process  approach 
to  effectively  deliver  management  ser¬ 
vices  to  meet  business  and  client 
requirements.  Its  process  improve¬ 
ment  can  be  managed  through  the 
CMMI  approach. 

•  CMMI.  A  methodology  enabling 
organizations  to  identify  the  maturity 
level  achieved  by  their  processes,  and 
to  design  and  implement  a  continuous 
improvement  plan  to  raise  their 
process  maturity  level  to  one  appropri¬ 
ate  for  their  business  objectives. 

Using  the  Standards  and 
Methodology 

For  this  article,  these  standards  relate  to 
organizational-level  quality  (e.g.,  what  is 
best  for  an  organization  or  its  sub-organi¬ 
zations),  not  just  lifecycle  processes  (e.g., 
what  is  required  to  perform  requirements 
analysis,  design,  or  testing).  For  instance, 
lifecycle  processes  normally  minimize  top 
management’s  business  goals  and  objec¬ 
tives  whereas  organizational-level  require¬ 
ments  (the  three  mentioned  ISO  stan¬ 
dards)  emphasize  business  needs,  goals, 
and  objectives.  In  my  opinion,  CMMI  ties 
together  organizational-level  and  lifecycle 
processes. 

Organizational  requirements  recognize 
the  need  for  owners  and  key  decision 
makers  to  decide  if  requirements  are  cost- 
effective,  an  organizational  need,  etc.  For 
example,  people  normally  recognize  the 
need  for  alternate  backup  sites  to  protect 
an  organization  from  collapse  due  to  a  dis¬ 
aster  at  a  key  organization  location. 
However,  at  the  organizational  level,  man¬ 
agement  may  determine  having  one  or 
more  backup  sites  is  too  expensive  since 
clients  are  unwilling  to  share  in  the  cost,  or 
the  organization’s  business  base  is  too 
diverse  in  functionality  and/ or  geographic 
location  to  have  back-up  sites.  Thus,  an 
organization  must  formally  identify  the 
risks  it  is  willing  to  accept  even  when 
attempting  to  be  certified. 

The  mentioned  standards  allow  an 
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Items 

ISO  27000:2005 

ISO  9001:2000 

ISO  20000:2005 

CMMI 

Planning 

1  Scope 

1 .2  Application 

1  Scope 

1 .2  Application 

Process  Area  (PA)  Project 
Planning 

Quality 

Management 

4  Information  Security 
Management  System 

4.2  Establishing  and 
Managing  the  ISMS 

4.2.1  Establish  the  ISMS 

4.2.2  Implement  and 
Operate  the  ISMS 

4.2.3  Monitor  and  Review 
the  ISMS 

4.2.4  Maintain  and 

Improve  the  ISMS 

4  Quality  Management 
System 

8.2.3  Monitoring  and 
Measurement  of 
Processes 

8.2.4  Monitoring  and 
Measurement  of 
Product 

5  Planning  and 

Implementing  New  or 
Changed  Services 

PA  -  Requirements 
Development 

PA  -  Integrated  Project 
Management  Specific 
Practices 

3.1  -3.5 

PA  -  Project  Monitoring 
and  Control  (PMC) 

General  Practice  (GP)  2.8 
Monitor  and  Control  the 
Process 

PA  -  Measurement 
and  Analysis  (MA) 

GP  3.1  Establish  a 

Defined  Process 

GP  3.2  Collect 

Improvement 

Information 

Quality  Plan, 
and 

Maintenance 
of  Documents 
and  Records 

4.3  Documentation 
Requirements 

4.3.2  Control  of 

Documents 

4.3.3  Control  of  Records 

4.2  Documentation 
Requirements 

4.2.2  Quality  Manual 

4.2.3  Control  of 

Documents 

4.2.4  Control  of  Records 

3.2  Documentation 

4.1  Planning  Service 
Management  (Plan) 

PA  -  Process  and  Product 
Quality  Assurance  (PPQA) 
PA  -  Configuration 
Management 

GP  2.6  Manage 

Configurations 

Audits 

6  Internal  ISMS  Audits 

8.2.2  Internal  Audit 

PA  -  PPQA 

Process 

Improvement 

8  ISMS  Improvement 

8.1  Continual  Improvement 

8.5  Improvement 

8.5.1  Continual 

Improvement 

4.4  Continual  Improvement 

PA-MA 

PA -PMC 

GP  2.2  Plan  the  Process 

Reviews 

7  Management  Review  of 
the  ISMS 

7.2  Review  Input 

7.3  Review  Output 

5.6  Management  Review 

5.6.2  Review  Input 

5.6.3  Review  Output 

4.3  Monitoring,  Measuring, 
and  Reviewing 

PA  -  Decision  Analysis 
and  Resolution 

GP  2.9  -  Objectively 

Evaluate  Adherence 

GP  2.10  -  Review  Status 
with  Higher  Level 
Management 

Table  1 :  Sample  Relationship  Showing  Similarities  Retween  the  Four  Standards 


organization  to  tailor  the  implementation 
of  the  standards  to  match  how  an  organi¬ 
zation  operates.  For  instance,  ISO  9001 
allows  clauses  to  be  deleted  if  an  organi¬ 
zation  does  not  implement  a  clause  (e.g, 
clause  7.3,  Design  and  Development,  if  an 
organization  does  not  design  or  develop 
products). 

But  how  does  an  organization  receive 
authorized  certification?  Is  this  process  of 
any  benefit  to  potential  clients? 

The  Process  to  Receive 
Certification 

Each  standards  development  group  has 
its  own  certification  process  and  there  are 
many  Internet  sites  discussing  the 
processes  to  receive  independent  certifi¬ 
cation.  Some  requirements  are  the  exis¬ 
tence  of  objective  evaluations  and  a  his¬ 
tory  (e.g.,  at  least  three  months)  of  arti¬ 
facts  (proof)  to  show  organizational 
processes  are  implemented.  Another 


requirement  is  for  the  organizational 
processes  to  comply  with  an  authorized 
standard.  Most  organizations  should  be 
able  to  ensure  this  requirement  is  being 
satisfied  through  an  objective  gap  analy¬ 
sis  (internal  audit)  of  their  processes  ver¬ 
sus  a  given  standard.  Many  organiza¬ 
tions — even  if  they  have  effective,  effi¬ 
cient  processes  in  place — find  they  lack 
objective  artifacts  showing  a  continuous 
and  objective  use  of  the  processes  stated 
within  a  given  standard. 

Conclusion 

The  identified  standards  have  publicly 
assessable  databases  with  information 
about  what  organizations  are  currently 
certified.  However,  clients  must  be  aware 
that  status  posting  may  take  weeks  to  be 
stored  into  a  database  or  for  an  organiza¬ 
tion  to  receive  a  formal  certificate. 
Because  of  this,  clients  must  determine 
the  cut-off  date  for  an  active  certificate 
(e.g.,  the  certificate  must  be  valid  on  the 


date  proposals  are  due,  so  many  calendar 
days  after  a  proposal  is  due,  or  at  the  time 
of  the  contract  award).  Another  option,  if 
an  organization’s  certificate  has  yet  to  be 
posted,  is  for  a  client  to  allow  an  organiza¬ 
tion  to  provide  a  copy  of  its  certification 
packet  to  indicate  the  certification  results. 
In  this  case,  the  RFP  needs  to  state  that  if 
an  official  certificate  is  being  processed 
that  the  entire  certification  packet  must  be 
included  in  the  proposal  so  a  client  can 
identify  the  auditor’s  recommendation  for 
approval.  In  this  situation,  I  recommend 
that  the  RFP  also  states  that  an  official 
certificate  must  be  provided  upon  con¬ 
tract  award. 

Whether  a  database  or  a  certificate 
copy  is  used,  clients  need  to  be  aware  that 
some  organizations  exaggerate  the  certifi¬ 
cation  results.  Commonly,  an  organiza¬ 
tion’s  subset  may  be  certified,  but  an  orga¬ 
nization  indicates  the  certification  is  at  the 
organizational  level  covering  all  organiza¬ 
tional  subsets.  To  overcome  this  problem, 
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certificates  and  certification  databases 
clearly  state  what  part  of  an  organization 
is  certified.  As  a  result,  clients  need  to  go 
beyond  just  accepting  the  word  of  organi¬ 
zations.  Clients  need  organizations  to  pro¬ 
vide  objective  evidence  (e.g.,  copy  the  cer¬ 
tificate)  or  have  the  client  verify  an  organi¬ 
zation’s  certification  statement  based  on 
certification  databases. 

Is  this  proof  of  certification  worth  it? 
Clients  can  use  certifications  to  establish 
die  chances  that  an  organization  or  subset 
can  deliver  the  needed  product  or  services 
on  time,  within  cost,  and  at  the  needed 
quality  level.  When  an  organization  pro¬ 
vides  proof  of  performance  and  a  copy  of 
its  certificate,  this  provides  a  client  with  a 
degree  of  confidence  that  the  organiza¬ 
tion  can  satisfy  the  client’s  needs. 

However,  an  important  reminder  for 
clients  is  diat  the  existence  of  a  certificate 
does  not  mean  an  organization  will  actual¬ 
ly  use  what  is  said  within  a  certificate.  As  a 
result,  clients  need  to  contractually  receive 
die  plans,  processes,  steps,  etc.,  used  to 
receive  an  organization’s  certificate(s),  and 
organizations  must  receive  client  approval 
for  modifications  to  these  plans,  etc. 

Having  performed  independent  verifi¬ 
cation  and  validation  (IV&V)  for  more 
dian  12  years,  I  have  seen  organizations 
win  contracts  based  in  part  on  certifica¬ 
tions  (e.g.,  having  a  CMMI  Level  5),  but 
diey  do  not  implement  these  features  dur¬ 
ing  a  contract.  In  diis  situation,  I  blame 
die  client  for  not  verifying  the  implemen¬ 
tation  of  what  was  promised  or  clearly 
implied  in  the  proposal.  For  example,  I 
have  seen  a  major,  well-known  organiza¬ 
tion’s  CMMI  Level  5  subset  (which  was 
stated  in  their  proposal  and  contract)  not 
be  penalized  for  failure  to  use  promised 
standards.  Thus,  the  client  promoted  the 
importance  of  cost  and  schedule  over 
quality. 

Therefore,  a  client  can  use  an  organiza¬ 
tional  certificate  to  show  an  organization 
has  implemented  documented  processes 
(that  were  based  on  known  standards). 
However,  it  is  up  to  the  client  to  some¬ 
times  require  an  organization  to  use  the 
certified  processes  for  a  given  contract. 

Also,  certification  does  not  guarantee 
successful  implementation  of  quality 
processes  or  delivery  of  quality  products 
or  services.  What  certifications  do  provide 
is  objective  evidence  that  a  certified  inde¬ 
pendent  group  has  examined  artifacts 
showing  that  an  organization  has  imple¬ 
mented  processes  to  satisfy  stated  stan¬ 
dards.  ♦ 
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Notes 

1.  Since  the  author  is  not  a  government 
employee,  he  is  not  providing  guidance 
currently  used  by  the  government  to 
assist  in  making  better  selections  based 
on  the  RFP  process. 

2.  The  U.S.  government  uses  RFPs  to  ask 
organizations  to  provide  a  proposal 
addressing  the  issues  provided  in  the 
model  contract  and  statement  of  work 
(SOW).  The  resulting  proposals  deter¬ 
mine  what  organization(s)  wins  a  con¬ 
tract  to  provide  the  SOW-stated  needs. 
Within  the  RFP,  the  government  iden¬ 
tifies  the  evaluation  criteria  (e.g., 
understanding  of  the  problem,  past 
performance,  technical  and/or  man¬ 
agement  approach,  and  cost)  and  the 
priority  or  weight  of  each  criterion. 

3.  The  standards  I  cite  do  not  provide 
detailed  requirements  (e.g.,  what  level 
of  software  testing  is  required).  They 
are  at  a  high  level  to  address  what 
organizations  need  to  implement  to 
ensure  quality  processes,  products,  or 
services,  without  disrupting  an  organi¬ 
zation’s  goals,  objectives,  and  level  of 
acceptable  risks. 
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Using  Software  Quality  Methods  to 
Reduce  Cost  and  Prevent  Defects 
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Everyone  knows  that  it’s  better  to  “do  it  right  the  first  time.  ”  But  in  organisations,  this  requires  the  ability  to  predict  out¬ 
comes  of  their  established  “best  practices”  as  well  as  the  ability  to  justify  costs  when  it  comes  to  applying  what  may  be  new 
approaches.  This  is just  as  true  in  software  development  as  it  is  in  any  other  business  practice.  This  article  will  survey  some 
of  these  best  practices  and  present  a  method  for  evaluating  the  costs  and  benefits  of  applying  them. 


Software  can  be  considered  a  product 
whose  production  is  fundamentally 
similar  to  other  products.  Improving  the 
quality  of  software  can  be  approached 
using  the  same  basic  principles  espoused 
by  quality  pioneers  such  as  W  Edwards 
Deming,  Philip  B.  Crosby,  and  Harold  F. 
Dodge.  These  principles  can  form  a  prac¬ 
tical  framework  for  ensuring  drat  appro¬ 
priate  requirements  are  set  for  software 
development  projects.  By  connecting 
established  software  engineering  practices 
to  the  objective  of  defect  prevention,  we 
can  apply  the  principles  of  quality  man¬ 
agement  to  software  development.  Using 
modeling  techniques,  it  is  possible  to  pre¬ 
dict  the  potential  cost  savings  and  defect 
reduction  expected. 

Quality  management  is  a  well-estab¬ 
lished  discipline  with  historic  roots  in 
manufacturing  industries.  W.  Edwards 
Deming  [1],  Philip  B.  Crosby  [2],  and  oth¬ 
ers  have  written  and  taught  extensively  in 
the  field.  The  classical  approach  to  quality 
management  can  be  summarized  in  these 
simple  steps: 

1.  Analyze  product  defects  to  determine 
root  causes. 

2.  Modify  processes  to  address  and 
remove  root  causes  of  defects. 

3.  Fix  defects  using  improved  processes. 

By  following  this  approach,  we  can 
realize  tire  goal  of  improving  product 
quality  by  removing  tire  causes  of  defects. 
As  Crosby  put  it:  “Quality  is  free.  It’s  not 
a  gift,  but  it  is  free.  What  costs  money  are 
the  non-quality  tilings — all  the  actions 
that  involve  not  doing  jobs  right  the  first 
time”  [2], 

Identifying  Best  Practices 

While  the  classical  approach  to  quality 
management  (as  taught  by  Crosby  and 
others)  would  suggest  that  each  organiza¬ 
tion  should  start  fresh  in  identifying  and 
fixing  process  defects  in  order  to  improve 
product  quality,  experience  suggests  oth¬ 
erwise.  The  famous  mathematician  and 
computer  scientist  Richard  Hamming 
once  asked,  “How  do  I  obey  Newton’s 
rule?  He  said,  ‘If  I  have  seen  further  than 


others,  it  is  because  I’ve  stood  on  the 
shoulders  of  giants.’  These  days  we  stand 
on  each  other’s  feet”  [3], 

If  we  want  to  profit  from  the  work  of 
pioneers  in  the  field  of  software  quality, 
we  owe  it  to  ourselves  and  them  to  stand 
on  their  shoulders.  This  means  that  we 
should  be  willing  to  adopt  proven  best 
practices  without  necessarily  requiring  that 
their  value  be  proven  first  in  our  specific 
development  environment.  We  will  try  to 
identify  some  of  these  best  practices  and 
focus  our  attention  on  them  here. 

Narrowing  Our  Focus 

There  is  no  doubt  that  the  quality  of  soft¬ 
ware  is  heavily  influenced  by  proper  atten¬ 
tion  to  every  phase  of  development,  from 
conceptual  design  through  requirements 
definition,  architecture,  detailed  design, 
construction,  testing,  documentation, 
training,  deployment,  and  sustainment. 
However,  for  tire  purposes  of  this  article, 
we  will  focus  on  what  Steve  McConnell 
refers  to  as  the  software  construction  [4]  phase 
of  software  development. 

Best  Practices  in  Software 
Construction 

This  article  will  treat  four  areas  of  best 
practices  in  software  construction: 

•  Uniform  coding  standards. 

•  Automated  Unit  Testing  (AUT). 

•  Root  cause  analysis. 

•  Code  reuse. 

These  areas,  in  turn,  can  be  linked  to 
die  four  software  construction  fundamen¬ 
tals  cited  in  the  IEEE  Computer  Society’s 
“Guide  to  the  Software  Engineering  Body 
of  Knowledge”  [5],  It  stated  that  the  fun¬ 
damentals  of  software  construction 
include: 

•  Minimizing  complexity. 

•  Anticipating  change. 

•  Constructing  for  verification. 

•  Standards  in  construction. 

Proper  attention  to  diese  areas  will 
lead  to  improved  quality  in  the  software 
we  create,  while  moving  closer  to  Crosby’s 
idea  that  quality  can,  in  fact,  be  free. 


Uniform  Coding  Standards 

Coding  standards  incorporate  experience 
and  best  practices  at  a  detailed  level  into 
the  software  construction  process. 
Typically,  these  include  the  seemingly  triv¬ 
ial,  such  as  spelling,  use  of  names,  and 
upper/lower  case;  the  moderate,  such  as 
die  code  matching  the  in-line  documenta¬ 
tion;  and  the  critical,  such  as  proper  man¬ 
agement  and  disposition  of  objects, 
exception  handling,  completeness  of 
branch  tests,  and  so  forth. 

The  use  of  uniform  standards  pro¬ 
vides  a  wide  range  of  benefits: 

•  Readability.  Any  programmers  writ¬ 
ing  to  the  same  standards  will  be  better 
able  to  read,  critique,  or  even  take  over 
software  written  by  others.  This  saves 
time  and  avoids  misunderstandings  in 
areas  including  peer  reviews,  updates 
and  maintenance,  and  reassignments. 

•  Support  by  tools.  Static  analysis  tools 
are  available  for  contemporary  pro¬ 
gramming  languages  and  environ¬ 
ments,  which  incorporate  die  ability  to 
check  for  adherence  to  coding  stan¬ 
dards  and  best  practices  in  writing 
code.  By  adopting  the  standards  sup¬ 
ported  by  diese  tools,  we  obtain  the 
advantage  of  increased  automated  tool 
use,  one  of  the  metrics  used  in  the 
System  Evaluation  and  Estimation  of 
Resources  -  Software  Estimating 
Model  (SEER-SEM),  as  referenced  in 
die  forthcoming  Using  the  Model  sec¬ 
tion. 

•  Peer  review  benefits.  The  peer 
review  process  is  enhanced  by  the 
adherence  to  uniform  coding  stan¬ 
dards.  Code  is  more  accessible  to 
potential  reviewers  and  less  time  is 
wasted  adapting  to  differing  approach¬ 
es.  The  review  can  focus  on  actual  and 
potential  defects  and  dieir  causes. 

In  addition  to  checking  for  adherence 
to  standards,  peer  review  leads  to  the  shar¬ 
ing  of  ideas  and  improved  coding  tech¬ 
niques.  Inspection  by  the  developer  prior 
to  review  may  contribute  to  defect  pre¬ 
vention  as  well. 

Government  audit  of  the  peer  review 
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process  is  enabled  by  coding  standards. 
Software  should  be  randomly  reviewed  on 
behalf  of  the  customer  in  order  to  ensure 
that  a  uniform  approach  is  being  followed. 

AUT 

Developers  have  long  been  responsible 
for  unit  testing  their  code.  This  involves 
testing  the  smallest  possible  part  of  a  pro¬ 
gram  to  ensure  correct  operation. 
Techniques  for  unit  testing  include  the  use 
of  a  debugger  to  step  through  a  routine, 
following  a  script  which  exercises  the 
desired  functions,  and  the  use  of  a  test 
harness  or  framework  to  execute  tests  auto¬ 
matically.  The  last  of  these  techniques  is 
generally  referred  to  as  AUT. 

While  strategies  for  analyzing  unit  test 
requirements  include  Dr.  Thomas  Radi’s 
TestGen  for  Ada  [6]  from  1989,  the  best- 
known  lineage  of  modern  AUT  dates 
from  SUnit  for  Smalltalk  [7]  in  1994.  This 
was  followed  in  1999  by  JUnit  (for  Java). 
Because  of  this  heritage,  and  the  fact  that 
die  basic  structure  of  these  test  harnesses 
has  been  carried  forward  into  multiple 
environments,  AUT  is  often  represented 
by  the  XUnit  family  of  test  harnesses, 
including  JUnit  and  NUnit  (for  Micro- 
soft.NET).  Note  that  the  automation  in  dais 
family  of  test  harnesses  is  in  the  execution 
of  tests.  There  are  other  tools  available 
diat  will  help  to  create  at  least  a  skeleton 
of  the  tests  themselves.  It  is  up  to  the 
developer  to  ensure,  by  the  use  of  tools 
and  inspection,  that  there  is  sufficient  cov¬ 
erage  of  input  values  and  paths  through 
die  code  to  provide  die  desired  thorough¬ 
ness. 

There  is  a  positive  impact  on  design 
when  automated  unit  tests  are  implement¬ 
ed  from  the  beginning.  In  order  to  prepare 
for  AUT,  code  must  be  designed  to  be 
tested.  Construction  techniques  such  as 
dependency  injection  or  dependency  lookup  [8] 
help  to  reduce  coupling  between  software 
modules,  enhance  modularity,  and  aid  in 
testability.  For  a  more  complete  discussion 
of  unit  testing  patterns  and  techniques, 
see  [8], 

In  addition  to  aiding  in  the  initial 
assurance  of  correct  operation,  automated 
unit  tests  serve  as  regression  tests  for 
existing  methods  and  routines  in  the 
course  of  development  by  continuing  to 
test  previously  working  code  each  time 
diey  are  run.  During  maintenance  and 
enhancement,  diis  regression  testing  helps 
to  prevent  the  introduction  of  new  errors 
into  existing  code. 

When  used  together  with  requirements 
for  code  coverage,  automated  unit  tests 
can  be  used  to  both  prevent  regression 
errors  and  set  minimum  standards  of  cor¬ 


rect  operation.  There  are  tools  available 
which  will  work  in  conjunction  with  unit 
test  tools  to  show  how  much  of  die  code 
under  test  is  being  executed  in  a  particular 
scenario. 

One  well-known  approach  to  AUT, 
Test  Driven  Development  [9],  extends  the 
testing  model  so  that  tests  are  written  first, 
dien  code  is  added  to  pass  the  tests.  This 
has  the  additional  benefit  of  focusing 
development  on  the  requirements  and  dis¬ 
couraging  what  has  been  called  feature  creep : 
adding  features  or  capabilities  while  pro¬ 
gramming. 

The  “Haves”  and  the  “Have-Nots” 

What  we  found,  in  an  informal  survey  of 
users  of  AUT,  is  diat  development  organi¬ 
zations  which  use  it  do  not  have  detailed 

“When  used  together 
with  requirements  for 
code  coverage, 
automated  unit  tests 
can  be  used  to  both 
prevent  regression  errors 
and  set  minimum 
standards  of  correct 
operation.  ” 

cost  comparisons  available.  In  general, 
once  they  started  using  it,  they  just  never 
went  back  to  traditional  manual  methods, 
nor  have  they  deemed  it  wordiwhile  to 
conduct  comparative  studies.  Those  who 
have  not  yet  adopted  these  tools  have 
sometimes  not  done  so  because  of  the 
perception  that  it  will  cost  more.  We  will 
attempt  to  show  that,  over  die  course  of 
development,  this  perception  is  false. 

Root  Cause  Analysis 

Root  cause  analysis  is  die  most  fundamen¬ 
tal  technique  of  quality  management,  and 
is  a  CMMI  Level  5  practice  area.  It  is 
important  to  use  this  technique,  however, 
regardless  of  the  CMMI  level.  Fixing 
defects  in  a  product  without  addressing 
die  cause  is  known  in  classic  manufactur¬ 
ing  environments  as  rework.  It  is  no  differ¬ 
ent  in  software  development,  where  we 
call  it  fixing  bugs.  Without  addressing  root 
causes,  there  is  no  reason  to  believe  that 
simply  reworking  software  defects  will 


improve  the  quality  of  die  overall  result, 
since  the  same  (potentially  flawed) 
processes  are  used  to  make  die  changes  as 
were  used  originally  to  write  die  code. 

Accepted  techniques  for  analyzing 
root  causes  include  die  5  whys  method  and 
Kepner-Tregoe  Problem  Analysis  method 

[10] ,  The  former  method  was  originally 
developed  at  Toyota  Motor  Corporation 
and  is  deceptively  simple:  When  analyzing 
a  defect  or  failure,  start  by  asking  why?,  and 
continue  asking  this  for  each  answer  until 
a  satisfactory  root  cause  is  reached.  The 
number  5  is  simply  a  guideline  in  this  case. 

Kepner-Tregoe’s  Problem  Analysis 
takes  a  different  approach,  asking  (with 
regard  to  a  defect  or  failure):  What?  Where? 
When ?  and  To  What  Extent?  These  ques¬ 
tions  are  addressed  in  terms  of  what  the 
problem  is,  what  it  could  be  (but  is  not), 
and  what  changes  and  differences  are 
associated  with  the  occurrence.  These  are 
then  analyzed  for  determining  possible 
root  causes. 

Analyzing  and  addressing  root  causes 
is  essential  to  improving  die  development 
process.  However,  in  order  to  preserve 
and  then  later  analyze  the  knowledge 
gained  by  this  approach,  it  is  necessary  to 
classify  root  causes. 

Classification:  Root  Cause  Taxonomy 

A  variety  of  schemes  have  been  proposed 
and  used  for  classification  of  root  causes. 
These  include  IEEE  Standard  1044  and 
IBM’s  Orthogonal  Defect  Classification 

[11] ,  However,  diese  do  not  lend  them¬ 
selves  well  to  automated  analysis. 

Boris  Beizer  [12]  provided  a  simple 
approach  to  root  cause  classification.  The 
Beizer  Taxonomy  yields  a  4-digit  number. 
Based  on  an  open-ended  hierarchical  clas¬ 
sification  scheme,  it  can  be  extended  widi- 
out  changing  die  original  categories. 

One  of  the  advantages  of  this 
approach  is  that  it  is  amenable  to  analysis 
using  database  query  tools,  Pareto  dia¬ 
grams,  and  statistical  techniques  for 
extracting  patterns  from  data.  Consistent 
use  of  this  taxonomy  can  provide  an 
enterprise  with  insights  into  areas  for 
process  improvement  that  might  not  be 
readily  apparent  otherwise.  The  enterprise, 
for  example,  may  be  a  customer  for  soft¬ 
ware  which  is  written  by  a  variety  of  devel¬ 
opment  organizations. 

Table  1  shows  the  top  level  categories 
of  die  Beizer  Taxonomy. 

Software  Reuse 

Through  the  use  of  uniform  coding  stan¬ 
dards  and  designing  software  to  be  readily 
tested  by  automated  unit  tests,  diere  is  an 
increased  likelihood  that  software  devel- 
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oped  for  one  project  or  program  can  be 
reused  by  another.  Good  documentation 
and  modular  design  are  also  needed  to 
make  software  reusable. 

Additional  value  can  be  added  when 
reliable  open-source  or  other  freely  avail¬ 
able  software  with  a  large  body  of  users 
can  be  applied  to  a  project.  Depending  on 
the  development  environment  and  lan¬ 
guage^)  involved,  this  may  mean  looking 
at  open-source  projects  or  libraries  such  as 
the  Microsoft  Enterprise  Library  (for  the 
.NET  Framework). 

Many  of  these  sources  meet  the  other 
previously  discussed  criteria,  such  as  using 
well-established  coding  standards  and 
including  automated  unit  tests.  Available 
software  which  does  not  meet  these  crite¬ 
ria  has  an  implicit  cost  in  adopting  it  for 
reuse:  Bringing  the  software  up  to  the 
same  standards  used  during  specific  devel¬ 
opment  for  the  project  may  be  required. 

Cost/Benefit  Analysis 

In  order  to  analyze  the  benefits  of  intro¬ 
ducing  techniques  aimed  at  improving 
software  quality,  we  need  to  find  a  way  to 
predict  the  results.  This  is  accomplished 
through  the  use  of  modeling  and  compar¬ 
ing  the  predicted  outcome  of  the  develop¬ 
ment  effort  under  varying  conditions  and 
practices. 

Modeling  the  Cost  of  Improving 
Quality 

Crosby  defines  the  cost  of  quality  as  “. . .  the 
expense  of  non-conformance — the  cost 
of  doing  things  wrong”  [2],  This  can  be 
added  up  after  the  fact  as  the  cost  of 
rework  and  scrapping  the  work  (in  the 
case  of  software,  the  cost  of  fixing  defects 
in  the  code).  But,  suppose  we  want  to  pre¬ 
dict  ahead  of  time  what  the  benefits  might 
be  of  applying  some  of  the  fundamental 
best  practices  previously  described  to  a 
development  project?  Take,  for  example, 
the  problem  of  the  benefits  of  AUT  and 
static  analysis. 

Costs  and  Benefits  of  AUT 

One  of  the  most  intractable  problems  in 
considering  the  introduction  of  best  prac¬ 
tices  into  the  software  construction  phase 
has  been  justifying  the  cost.  If  you 
attempt  to  look  at  the  value  of  AUT  in 
isolation,  this  fundamental  problem  pre¬ 
sents  itself:  Most  sources  agree  that  to  test 
n  lines  of  source  code  requires  at  least  n  to 
n  +  25  percent  additional  lines  of  code 
[13].  If  you  apply  this  to  a  traditional  esti¬ 
mating  approach  which  multiplies  source 
lines  of  code  (SLOC)  by  hours  or  dollars, 
it  will  appear  that  the  use  of  this  technique 
will  add  significant  cost  to  the  project.  If 


this  were  so,  then  the  apparent  fact  that 
organizations  using  this  technique  are  so 
thoroughly  committed  to  it  would  seem  to 
be  contradictory. 

Fortunately,  there  is  a  more  compre¬ 
hensive  approach.  The  cost  modeling  soft¬ 
ware  which  is  in  use  by  the  AF  Electronic 
Systems  Center  at  Hanscom  AFB  takes  in 
to  account  a  number  of  factors  in  addition 
to  SLOC.  These  include  the  use  of  auto¬ 
mated  tools,  the  degree  of  testing,  and  the 
extent  of  quality  assurance.  This  allows  us 
to  see  past  the  SLOC  issue,  and  estimate 
the  savings  which  are  possible  by  the  use 
of  these  techniques. 

Using  the  Model 

The  commercially  available  SEER  for 
Software  application  is  a  comprehensive 
software  project  estimation  system. 
SEER-SEM  is  the  core  estimation  capa¬ 
bility  originally  based  on  the  effort  and 
schedule  relationships  developed  by  Dr. 
Randall  Jensen  [14].  The  SEER-SEM 
product  comes  with  a  comprehensive  set 
of  knowledge  bases  which  offer  default 
parameter  values  that  target  complexity 
and  productivity  factors  for  a  wide  variety 
of  project  types.  These  knowledge  bases 
are  developed  and  tested  by  analyzing 
thousands  of  projects.  Initial  inputs  to  a 
SEER-SEM  estimate  include  a  descrip¬ 
tion  of  the  platform,  application  type, 
reuse  scenario,  development  methods, 
and  development  standards.  Detailed 
inputs  include  several  ways  to  enter  soft¬ 
ware  project  size  as  well  as  several  pro¬ 
ductivity-related  parameters  that  help 
describe  the  people  developing  the  soft¬ 
ware,  the  methods  and  tools  used,  the 
customer-driven  requirements  and  con¬ 
straints,  and  the  system  being  developed. 
This  allows  the  user  to  do  n>hat-if?  analy¬ 
sis  based  on  a  variety  of  development 
strategies  using  various  parameters  relat¬ 
ed  to  the  size  of  a  project,  its  difficulty, 
the  experience  of  the  developers,  and  the 
tools  and  techniques  used. 

The  use  of  a  cost  modeling  tool  to  do 
what-if?  studies  serves  as  a  means  to  simu¬ 
late  different  scenarios.  Ideally,  it  can  pro¬ 
vide  an  objective  assessment  of  how  cost, 
schedule,  and  quality  might  change  as  pro¬ 
ject  assumptions  change.  In  using  SEER- 
SEM,  you  can  evaluate  the  impacts  of 
project  assumptions  to  the  whole  project, 
not  just  the  construction  phase  that  is 
most  directly  impacted  by  AUT  and  static 
analysis  tools. 

From  the  perspective  of  cost  model¬ 
ing,  AUT,  along  with  tools  that  check 
source  code  for  syntax  or  security  errors, 
fall  into  the  general  category  of  automated 


Top-level  categories: 

•  Oxxx  Planning 

•  Ixxx  Requirements  and  Features 

•  2xxx  Functionality  as  Implemented 

•  3xxx  Structural  Bugs 

•  4xxx  Data 

•  5xxx  Implementation 

•  6xxx  Integration 

•  7xxx  Real-Time  and  Operating 
System 

•  8xxx  Test  Definition  or  Execution 
Bugs 

•  9xxx  Other 

Table  1:  Top-Level  Categories  of  the  Beider 
Taxonomy 

tool  use.  According  to  the  SEER-SEM 
model  [15],  increasing  the  use  of  automat¬ 
ed  tools  actually  decreases  (rather  than 
increases)  the  cost  of  developing  software. 
In  addition,  it  reduces  the  number  of 
defects  expected  to  be  produced  by  the 
process. 

In  one  example,  changing  the  model 
parameter  Automated  Tool  Use  from 
Nominal  to  High,  resulted  in  a  projected 
decrease  in  effort  of  9  percent,  accompa¬ 
nied  by  a  decrease  of  13  percent  in  pre¬ 
dicted  defects.  This  shows  that,  contrary 
to  a  cursory  estimate,  doing  the  extra  work 
to  develop  automated  unit  tests  (along 
with  other  automated  tool  use)  can  be 
expected  to  reduce  the  overall  effort 
involved  in  software  development.  While 
this  result  appears  interesting,  it  is  impor¬ 
tant  to  understand  that  changing  a  single 
parameter  to  study  the  cost  and  quality 
trade-off  of  AUT  can  be  viewed  as  overly 
simplistic.  Fortunately,  there  are  other 
dimensions  to  this  scenario  that  can  be 
evaluated  using  a  cost  modeling  tool. 

It  is  fair  to  say  that  introducing  auto¬ 
mated  tool  use  into  a  development  organi¬ 
zation  will  not  produce  instant  benefits. 
Fortunately,  the  cost  modeling  tools  allow 
for  a  more  nuanced  look  at  this  ivhat-if  ? 
scenario.  In  addition  to  looking  at  the 
impact  of  automated  tool  use  improve¬ 
ment,  we  can  consider  experience  factors 
and  the  potential  for  added  volatility. 

As  an  example,  we  will  examine  a  pro¬ 
ject  with  three  major  applications  and  two 
vendor-supplied  applications.  The  project 
is  of  moderate  criticality  in  terms  of  the 
overall  specification,  quality  assurance, 
and  test  levels  required.  There  are  three 
cases  examined: 

•  Baseline:  Assumes  no  AUT,  which 
notionally  represents  the  organization 
as-is.  The  team  has  nominal  experience 
with  the  development  environment, 
tools,  and  practices. 

•  Introducing  AUT:  Takes  the  baseline 
scenario  with  the  introduction  of 
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Baseline 

Introducing 

AUT 

Difference 

AUT  + 
Experience 

Difference 

Schedule  Months 

17.09 

17.41 

2% 

16.43 

-4% 

Effort  Months 

157 

166 

6% 

139 

-11% 

Hours 

23,881 

25,250 

6% 

21,181 

-11% 

Base  Year  Cost 

$2,733,755 

$2,890,449 

6% 

$2,424,699 

-11% 

Defect  Prediction 

84 

81 

-4% 

68 

-19% 

Table  2:  Cost  Model  Trades 


AUT.  This  will  result  in  a  small 
increase  of  the  automated  tool  use 
parameter  as  well  as  the  modern  devel* 
opment  practices.  However,  since  the 
use  of  these  tools  is  new  to  the  orga¬ 
nization  and  teams,  there  is  a  decrease 
in  the  overall  development  environ¬ 
ment  experience.  Also,  introduction  of 
new  tools  may  inject  some  volatility 
into  the  system.  This  is  because  the 
team  may  need  to  tweak  the  process  to 
accommodate  the  tools  being  used. 
For  example,  they  may  need  to 
upgrade  to  the  latest  service  packs  for 
their  operating  system  or  development 
environment  in  order  to  integrate  the 
unit  test  tools  effectively. 

•  Introducing  AUT  and  Added 
Experience:  Similar  to  the  previous, 
but  with  the  caveat  that  the  team  has 
had  some  training  in  the  use  of  the 
AUT  tools  and  has  established  the 
methods  used  as  routine.  This  training 
may  be  done  in  a  traditional  classroom 
or  boot  camp  environment,  or  it  could 
be  on-the-job  training.  In  either  case, 
the  assumption  here  is  that  the  team 
has  gained  some  experience  in  using 
the  AUT  tools  and  tire  process  is  well 
integrated  into  their  overall  develop¬ 
ment  process. 

The  results  of  these  three  runs  are 
shown  in  Table  2. 

Results  include  the  estimated  schedule 
months  or  duration  and  the  estimated 
effort  expressed  as  effort  months,  hours, 
and  cost.  The  last  row  in  the  table  is  the 
estimated  number  of  delivered  defects. 
The  defect  estimate  in  SEER-SEM  takes 


into  account  the  project  size,  program¬ 
ming  language  used,  as  well  as  many  of 
the  productivity  factors  used  to  estimate 
effort  (e.g,  requirements  definition  for¬ 
mality,  specification  level,  test  level,  and 
others). 

The  results  of  this  analysis  demon¬ 
strate  that  there  is  an  initial  hit  to  overall 
productivity  when  introducing  new  tools 
and  methods.  However,  this  impact  is  not 
a  long-term  change,  but  rather  a  short¬ 
term  setback  that  can  be  overcome  by 
training  or  general  experience.  It  is  worth 
noting  that  even  without  the  benefit  of 
experience,  the  number  of  defects  went 
down  with  the  introduction  of  AUT. 

By  adding  the  dimension  of  defect 
prediction  into  the  cost-modeling  method, 
you  can  quantify  the  impact  of  changes  in 
tools,  methods,  and  staff  capabilities  used 
on  a  project,  not  just  in  terms  of  invest¬ 
ment  or  savings,  but  also  in  terms  of 
improved  quality.  Software  managers  need 
to  be  able  to  justify  investments  in  new 
tools  and  technologies,  but  using  claims  by 
tool  vendors  can  be  misleading.  Invest¬ 
ment  in  quality  improvements  should  be 
analyzed,  not  just  for  the  general  effects, 
but  for  their  effects  on  specific  projects.  It 
is  important  to  not  just  look  at  the  benefit 
of  the  coding  effort  (as  many  tool  vendors 
will  provide),  but  to  the  overall  benefit  of 
the  project. 

In  addition  to  the  end  result,  visibility 
into  the  defect  profile  over  the  develop¬ 
ment  period  is  available  as  part  of  this 
cost  model.  The  defect  prediction  is  consid¬ 
ered  to  be  defects  delivered  at  the  end  of 
development.  However,  projects  find  and 


remove  many  more  defects  during  devel¬ 
opment.  Every  project  has  a  defect  potential 
that  represents  the  opportunity  for 
defects  to  occur  during  development  and 
beyond.  The  defect  potential  is  based  on 
size,  complexity,  and  other  factors.  In 
general,  most  of  the  potential  defects  are 
found  and  removed  through  the  develop¬ 
ment  process.  However,  not  all  are 
removed,  leaving  delivered  defects.  The 
percentage  of  defects  removed  during 
development  is  called  the  defect  removal  effi¬ 
ciency.  This  is  calculated  as  the  total 
defects  removed  divided  by  total  defect 
potential.  Higher  defect  removal  efficien¬ 
cies  are  typically  associated  with  the  use 
of  more  rigorous  or  formalized  software 
development  methods. 

The  detail  behind  the  quality  metrics  in 
this  analysis,  shown  in  Table  3,  is  provided 
by  the  cost  model.  When  introducing 
AUT,  you  see  a  small  increase  in  the  defect 
removal  efficiency.  However,  this  increase 
is  offset  by  an  increase  in  the  overall 
defect  potential  that  results  in  an  increased 
number  of  hours  spent  removing  each 
defect.  However,  when  you  couple  AUT 
with  the  requisite  experience,  the  increase 
in  defect  removal  efficiency  is  boosted  by 
the  fact  that  the  overall  defect  potential  is 
reduced.  This  reduction  in  defect  poten¬ 
tial,  combined  with  the  overall  effort 
reduction,  quantifies  the  intuitive  adage 
that  the  cheapest  defect  to  remove  is  an 
avoided  defect. 

While  cost  modeling  tools  have  been 
used  for  budgeting  and  proposal  purpos¬ 
es,  they  can  be  employed  as  a  strategic  tool 
to  evaluate  how  changes  in  processes  and 
methods  will  impact  a  software  develop¬ 
ment  organization.  Cost  modeling  tools 
provide  a  tangible  method  for  understand¬ 
ing  how  the  use  of  new  methods  and  tools 
can  impact  cost,  schedule,  and  quality.  In 
this  case,  it  was  demonstrated  that  invest¬ 
ment  in  quality  methods  is  justified. 
Additional  benefits  can  be  obtained  when 
looking  at  required  maintenance  efforts. 
Having  fewer  defects  means  that  less  time 
is  spent  fixing  problems,  giving  more  time 
and  resources  to  improving  the  system. 

Summary 

Adopting  and  enforcing  best  practices  in 
software  construction  leads  to  better 
results  at  a  lower  cost.  The  practices  out¬ 
lined  in  this  article  are  a  good  starting 
point  for  a  quality  improvement  program 
in  the  construction  phase  of  software 
development.  These  best  practices  can  be 
implemented  directly  by  a  development 
organization,  or  incorporated  into  con¬ 
tractual  requirements  by  an  acquisition 
organization.  Modeling  tools  can  be  used 


Table  3:  Defect  Prediction  Detail 


Baseline 

Introducing 

AUT 

Difference 

AUT  + 
Experience 

Difference 

Potential  Defects 

738 

756 

2% 

668 

-9% 

Defects  Removed 

654 

675 

3% 

600 

-8% 

Delivered  Defects 

84 

81 

-4% 

68 

-19% 

Defect  Removal 
Efficiency 

88.6% 

89.3% 

89.8% 

Hours/Defect 

Removed 

36.52 

37.41 

2% 

35.30 

-3% 
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to  demonstrate  the  cost  effectiveness  of 
implementing  best  practices,  and  to  help 
justify  any  initial  cost  to  the  development 
organization  in  instituting  these  prac¬ 
tices. ♦ 
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Data  (mis)Management 


While  I  wrote  this  column,  I  was  at  an  altitude  somewhere 
around  34,000  feet,  flying  from  Albuquerque  to  Houston. 
I  was  working  on  a  briefing  I  will  be  presenting,  and  started  look* 
ing  for  a  support  file  that  had  critical  information  (translation:  I 
am  looking  for  a  Dilbert  cartoon  I  used  in  a  previous  briefing). 
As  I  search  and  search  unsuccessfully,  I  started  thinking  about 
the  large  amount  of  data  that  I  carry  with  me,  and  the  entirety  of 
my  “electronic  life.” 

The  1980s:  I  was  running  the  Control  Program  for 
Microcomputers  (also  known  as  CP/M),  then  the  Commodore 
OS,  then  MS-DOS  2.11.  My  “life”  consisted  of  no  more  than  a 
dozen  160,  dien  180,  and,  later,  360KB  5  1 /4-inch  floppies.  I 
probably  carried  100-200  files  with  me  when  I  changed  jobs. 

The  1990s:  UNIX,  SunOS,  Windows  3.1,  Macintosh  OS 
Version  4.0,  and  Windows  95.  My  “electronic  life”  could  be  car¬ 
ried  on  several  boxes  of  1 ,4MB  floppies,  or  maybe  an  eight  mil¬ 
limeter  tape,  and  eventually  a  few  CDs.  I  actually  found  “Backup 
of  Dave  Cook’s  life”  CDs  from  1997,  when  I  retired  from  the 
AF.  Two  CDs  contained  all  that  I  felt  worthy  of  keeping:  1,344 
files,  taking  up  about  1GB. 

Now:  Windows  XP,  and  then  Vista.  I  carry  a  250GB  portable 
hard  drive  when  I  travel,  plus  a  few  8GB  thumb  drives  with  crit¬ 
ical  files.  A  complete  backup  of  my  “electronic  life”  (minus  the 
music  and  videos)  takes  12.5GB,  and  spans  11,218  files.  Of 
course,  that  does  not  include  8GB  that  my  more  than  11,000  pic¬ 
tures  takes  up.  Nor  does  it  include  the  7GB  of  music  (1,334 
songs)  and  154GB  of  videos  (27  movies  that  I  will  definitely 
watch  ...  someday!)  If  I  converted  all  of  this,  it  would  take  286 
CDs,  or  almost  130,000  of  the  1.44MB  floppies.  Wow! 

And,  of  course,  it  is  becoming  increasingly  difficult  to  find  a 
single  file  when  I  need  something.  I  tried  organizing  my  music 
into  “Artist”  folders.  But  then  I  ended  up  with  “Misc,”  “Misc 
from  my  daughter,”  “Comedy,”  etc.  It’s  the  same  with  business 
files.  I  started  out  with  Word  documents  and  PowerPoint  pre¬ 
sentations  ...  then  I  realized  that  some  should  be  grouped  under 


a  specific  customer  ...  then  some  are  sort  of  miscellaneous,  based 
on  a  presentation  or  work  from  the  past.  I  tried  organizing  them 
by  date  (“Files  from  1998”),  and  then  by  job  (“Stuff  from  STSC 
1997-2003”). 

And  now  we  have  distributed  data  to  worry  about.  I  have  an 
office  machine,  a  home  machine,  a  laptop  that  I  travel  with,  and 
a  spare  laptop  I  keep  in  the  living  room  (so  I  can  surf  and  watch 
TV  at  the  same  time!).  I  have  to  occasionally  worry  about  sync¬ 
ing  my  office  and  traveling  machine.  I  often  work  on  files  at 
home.  I  try  synchronizing  everything,  but  occasionally,  I  have  to 
panic  and  search  frantically  before  a  trip  to  find  the  latest  copy  of 
something.  I  DO  have  a  process:  I  try  to  remember  to  send  an 
updated  file  to  myself  (from  home)  in  an  e-mail,  and  I  use  vari¬ 
ous  tools  in  trying  to  keep  the  data  “in  sync.”  I  “officially”  use 
my  office  machine  as  my  “main”  machine  (and  just  clone  a  vir¬ 
tual  volume  to  my  laptop  when  I  travel).  The  problem  then 
revolves  back  to  finding  a  single  file  on  one  computer.  And  that’s 
where  the  current  suite  of  tools  occasionally  fails  me. 

Nothing  works  perfectly.  With  Vista,  I  can  just  type  in  a 
phrase  I  want  to  locate  (“SSTC  2006”)  and  get  lots  of  hits — none 
of  which  really  helped  me  find  that  perfect  Dilbert  cartoon.  Oh 
well,  I  have  lots  of  Dilbert  cartoons  to  choose  from  (173  in  the 
“Dilbert”  folder).  Luckily  for  me,  they  all  have  helpful  names 
(“Dilbert  Cartoon  June  1999”).  Let’s  just  face  the  facts:  It’s  get¬ 
ting  harder  and  harder  to  organize  and  arrange  data  so  that  you 
can  easily  find  what  you  need. 

Of  course,  I  am  sure  you  already  know  the  BEST  way  to 
manage  your  data,  right?  If  not,  let  me  explain  how  to  do  it  the 
RIGHT  way  so  that  you  don’t  lose  anything!  Wait  ...  hold  on  ...  I 
wrote  it  down  ...  I’ve  just  got  to  find  the  document  on  my  com¬ 
puter  ...  I  know  it’s  here  somewhere  ... 

— David  A.  Cook,  Ph.D. 

The  AEgis  Technologies  Group,  Inc. 

<dcook@aegistg.com> 
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